Ramblings of a Unix Geek

Post Quantum Cryptography Apache TLS settings

Sun, 07 Jun 2026 11:37:57 -0400

In December I documented TLS settings for Debian 13. These settings gave us TLS1.3, the ability to use ECDSA, backwards compatibility with some older software, and an A+ rating in SSL Labs.

Last week I re-ran the tests and noticed an additional flag had been added; “This server supports PQC (Post-Quantum Cryptography) key exchange.”

So I thought I’d summarize all the previous posts into a single one and describe my setup. There’s not really any new information here, but it means you don’t have to go back through 4 or 5 other posts to get the same stuff!

My Setup

For various complicated reasons I don’t use certbot to get my certificates from LetsEncrypt. Instead I use a shell script called Dehydrated. This can do http-01 or dns-01 challenges (in my case I use dns) and retrieve the certificates to a local file. From there I use ansible to deploy them (web servers, SMTP, IMAP, NNTP, MQTT,… lots of places across multiple machines!)

Getting RSA and ECDSA certs from LetsEncrypt

I want to use both RSA and ECDSA certificates on this site but Dehydrated can only handle one type at a time. Fortunately it has a work around; you can create a second certificate directory and have that override the cert type.

So in my case I configured the app to default to rsa and then created an override.

   % cat domains.txt
   *.sweharris.org sweharris.org > sweharris
   *.sweharris.org sweharris.org > sweharris_ecdsa

   % grep KEY_ALGO config
   KEY_ALGO=rsa

   % cat certs/sweharris_ecdsa/config
   KEY_ALGO="secp384r1"

Yes, I use a wildcard cert; it’s easier than managing a SAN entry of a dozen entries and then have to regenerate a new cert each time I add a new machine!

With this setup when I run the “refresh” command (dehydrated -c) it will place the RSA cert in the sweharris directory and ECDSA cert in the sweharris_ecdsa directory:

   % openssl x509 -noout -text -in certs/sweharris/cert.pem | grep Public.Key.Algo
               Public Key Algorithm: rsaEncryption

   % openssl x509 -noout -text -in certs/sweharris_ecdsa/cert.pem | grep Public.Key.Algo
               Public Key Algorithm: id-ecPublicKey

Inside the certs directory we will find four files of interest. These are always symbolic links to the latest version, so that provides a consistent name for us to use:

cert.pem
This is the public key for the server certificate
privkey.pem
This is the private key for the server.
chain.pem
This file contains the “chaining” certificates needed to generate a chain of trust to the root.
fullchain.pem
This file contains the chaining certificates and the server public key. Sometimes this can be easier to use.

We will use the fullchain and privkey files with Apache.

Configuring apache

Debian 13 comes with Apache/2.4.67 and OpenSSL 3.5.6. This supports TLS1.3.

I enabled the default-ssl site but commented out the SSLCertificateFile and SSLCertificateKeyFile lines. This is probably not necessary but it works :-)

TLS settings

I put the TLS settings in a conf-enabled file, at the top level, not restricted to a virtual host. In this way every vhost gets the same certificates and same ciphers.

SSLCertificateFile /etc/apache2/SSL/sweharris/fullchain.pem
SSLCertificateKeyFile /etc/apache2/SSL/sweharris/privkey.pem

SSLCertificateFile /etc/apache2/SSL/sweharris_ecdsa/fullchain.pem
SSLCertificateKeyFile /etc/apache2/SSL/sweharris_ecdsa/privkey.pem

SSLCipherSuite TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

Why those ciphers?

I go into more details in my previous posts (here and here) but to summarize:

These four are the only “fully” safe ones to use with TLS1.2 with RSA:

  TLS_AES_128_GCM_SHA256
  TLS_AES_256_GCM_SHA384
  TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

And a couple to enable ECDSA:

  TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

Unfortunately this will stop a few machines from connecting, so we add a couple of theoretically weak CBC ciphers:

  TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
  TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

And then we get a couple of nice new ones:

  TLS_CHACHA20_POLY1305_SHA256
  TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256

Virtual host settings

The virtual host setting is pretty generic; it’s pretty much the same as you’d set up for any virtual host.

<VirtualHost _default_:443>
  SSLEngine On
  ServerName mytesthost.sweharris.org
  ServerAlias mytesthost.sweharris.org

  DocumentRoot /var/www/ssl-docs
  <Location />
    Options +Indexes
  </Location>

  <Directory "/var/www/ssl-docs">
    Options Indexes
    Require all granted
  </Directory>

  ErrorLog ${APACHE_LOG_DIR}/ssl_error_log
  CustomLog ${APACHE_LOG_DIR}/ssl_access_log combined
  LogLevel warn
  CustomLog ${APACHE_LOG_DIR}/ssl_request_log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>

You can define multiple virtual hosts the same way. If you want the vhosts to have different TLS settings then move the relevant lines inside the VirtualHost setting. This may require replication of configuration, but you get more flexibility that way.

Other settings

Since we’re building a good configuration we can also (in the conf-enabled file) add a few more useful settings:

ServerSignature off
ServerTokens Prod

Header set Strict-Transport-Security "max-age=31536000; includeSubDomains"
Header set X-Content-Type-Options "nosniff"
Header set X-XSS-Protection "1; mode=block"
Header set X-Frame-Options "SAMEORIGIN"
Header set Referrer-Policy "no-referrer"
Header set Permissions-Policy: interest-cohort=()
Header set Content-Security-Policy: "script-src 'self'"

Some of these may definitely be VirtualHost specific in a larger environment, especially the Content-Security-Policy header!

Results.

An A+ rating

The first result is we get an A+ rating and supports PQC:

The Ciphers

The ciphers all support forward secrecy:

Notice the TLS1.3 ciphers have X25519MLKEM768 against them; these are our post-quantum ciphers and are the hybrid of X25519 and ML-KEM-768 (aka Kyber-768). Now this is new; 6 months previously this configuration only showed “ECDH x25519” as the cipher, so either openssl has improved and I got those improvements as a Debian patch, or SSL Labs has improved their detection (or both; could always be both!).

Weak CBC

We can see the two CBC ciphers have been marked as “Weak”. Now this is debatable. CBC isn’t necessarily weak, but it’s historically had a lot of bad implementations, leading to variations of POODLE attacks (amongst others). It’s was kinda expected that there will be yet another variation of POODLE on CBC, so SSLlabs flag these ciphers as weak, as an encouragement to move people off and onto stronger ciphers.

Without those two ciphers the following clients can’t negotiate nor talk to the server, and it’s been years since the last known attack on these ciphers, so I’m not concerned, and it doesn’t reduce the score and lets you support older clients (how many Java 8 apps are still out there? probably too many!)

    IE 11 / Win 7
    IE 11 / Win 8.1
    IE 11 / Win Phone 8.1
    IE 11 / Win Phone 8.1 Update
    Java 8u161 
    Safari 6 / iOS 6.0.1 
    Safari 7 / iOS 7.1
    Safari 7 / OS X 10.9
    Safari 8 / iOS 8.4
    Safari 8 / OS X 10.10

Security Headers

And, finally, the security headers score:

Summary

In theory I’m not really concerned about post-quantum. Definitely not for this site (I use TLS mostly for data integrity) but also in the wider cryptographic sense; IMHO practical useful quantum compute is still 20 years away, and has been for the past 20 years. I don’t see 2048 bit RSA from being broken in my lifetime.

BUT, and this is a big “but”, your customers might. So we need to understand how to do this properly.

Fortunately we kind of get this “for free” with a well configured server. I didn’t need to make any changes to my configuration in order to get this result.

The challenges, now, are client-side. According to the SSL Labs simulation the only clients currently supporting PQC are Chrome 131+, Firefox 135+, Edge 131+, and Opera 117+. It looks like we need the new Java 27, which is not yet on general release, to have support this cipher!

What Linux distribution should I use?

Wed, 22 Apr 2026 14:20:58 -0400

I got asked a question… this gives me a chance to write an opinion. I have lots of them!

What Linux distribution should I use?

Oh boy, this is a biggie and could almost be a religious one!

In the past we had the Unix wars and then the Linux distro wars. But with Microsoft making Windows worse and people trying out (and liking!) Linux is becoming a viable desktop operating system. Heck, even gamers have started to use Linux, with the March 2026 steam survey showing that 5% of steam users are using Linux. 5% sounds low, but this is just for gaming; even I (a Linux user for 30+ years) have a Windows machine to play games.

So if this is going to be the year of Linux on the desktop then what’s the right choice?

And it depends.

Are you learning Linux because you want it for job?

If you’re in North America then RedHat Linux is probably the number one OS you’ll find in the commercial space. Definitely in finance, which is where I’ve been. So if your goal is to learn skills that will help you find a job (or get promoted) then a RedHat or derivative is a good starting point. It’s not the best desktop, but it makes a really good server OS. You can also run KVM based virtualization and have virtual machines and networking which will let you experiment; want to try something out? Build a VM, play in it, destroy it when you’re finished.

The three main options, here, are Rocky Linux, Alma Linux or even a free RedHat developer trial. The RedHat solution is something you have to renew every year but it gives you real licenses for real RedHat and gives you access to their knowledge base.

These are Long Term Support (LTS) operating systems; if you install a 10.x release today it will still be supported until end May 2035. Upgrades between major versions has historically been a bit of pain, but once every 10 years? Not too bad!

The one big downside to RedHat is that they’ve become a little more hostile to the free derivatives in recent years; they can’t stop them from existing but they can make it harder. It’s for this reason that I have been migrating my own servers away from Rocky and onto Debian.

For home lab

If you don’t need to learn the RedHat way then you might want to look at Debian Linux instead. This is what I’m mostly using, these days.

Debian has many of the same advantages of RedHat except it’s not what large corporate entities tend to be using. It’s also a really good server OS, and can do the same KVM virtualization as RedHat. You could have both worlds, potentially; have a primary Debian OS and run some RedHat VMs to learn the corporate tools.

Debian’s lifecycle is a new release every 2 years, with each release supported for 3 years, and LTS support up to 5 years. That seems short compared to RedHat, but Debian has a smoother upgrade path between major versions; I have a machine that was originally installed with Debian 11, then upgraded to 12, and is now running 13. It was mostly smooth with only minor issues.

There are forks and derivatives of Debian, most notably Ubuntu, but I don’t see any reason to run these for a home server setup. There are some corporates using Ubuntu, but it’s not something I’d recommend.

And containers?

This kinda almost doesn’t matter. Generally you shouldn’t treat the container base as an OS to be used, but merely as a base layer.

The idea behind a container is that you layer your software on top of a base image and you don’t admin the OS layer. If you need to make a change (e.g. there’s patches needed) then you rebuild and redeploy. So you typically don’t need to understand the underlying OS beyond knowing dnf or apt.

Commonly you’ll find Debian or Alpine being used; historically I also saw a fair bit of Ubuntu but that seems to have dropped (at least from what I’ve seen). RedHat tried to get into the game but I’ve not seen anyone use it!

For desktops or gaming?

Now we’re getting into a large messy area. I, personally, use Debian with LXDE as my desktop environment. This is stable and means all the tools I use on my server can also be used on my desktop. However Debian, being a stable release, doesn’t always have the latest and greatest libraries and so some software may not work or some new hardware will have issues (back in 2022 I bought a “stick computer” which had an Intel Celeron J4125 CPU in it; Debian just didn’t work with the sound chip; Ubuntu did; might be interesting to see how things work today!)

Outside of that there’s really the two main families; those in the RedHat side of things and those from the Debian side of things.

Fedora and its remixes are probably the main RedHat based desktops. It has a very fast release cycle (roughly every 6 months) which means you might spend a fair bit of time upgrading. Fedora can be upgraded between major versions and even allows you to skip a version, but you’ll still be forced to upgrade once a year. The plus side is you’ll always have more recent libraries and newer versions of software compared to Debian.

The biggest Debian descendent used to be Ubuntu. It’s still popular. Note that it sometimes does stuff differently and pushing agendas (e.g. Wayland instead of X11; heavy use of snap; LXD containers) and it sometimes feels bloated. I haven’t used this for a few years because I really didn’t like how it felt. But that’s a personal preference.

In the past I’ve also used Linux Mint (Debian Edition), especially on low under-powered laptops. As I understand it, Mint is generally built on Ubuntu but LMDE is built on Debian. You’re meant to get the same experience on both; I just picked LMDE because it was closer to Debian.

And then we get into the weeds. If we look at DistroWatch then we’ll also find distros that have been optimized for gaming, such as CachyOS or Pop!OS. And more. I really can’t talk to this because I don’t really use Linux for gaming. I can only suggest testing them out and seeing what you like the most, or what works best on your hardware (different kernels, different libraries; you may get different performance).

Steam for Linux appears to be mostly focused on Ubuntu, but a many of these distros have it packaged in their app store (e.g. CachyOS).

What about Arch or Gentoo?

I consider these distros as doing Linux on “hard mode”.

I started with PCs in 1987 and learned how the machine booted all the way from power on through the BIOS to the hard disk boot loaders (primary, secondary) and so on. This meant that when Linux had its own bootloader (LILO) I understood what it was doing. Similarly I’ve spent more than enough time doing things like ./configure && make && make test && make install in the past (and even built my own distro in 1993 for deployment at sea). I don’t need to have my OS installer do this for me.

So, yes, it can be instructive to use these distros. You can get a lot closer to the guts of the OS and it may be worth testing them out in a VM to see what you can learn. But I wouldn’t use them daily.

On the plus side, the Arch wiki has some of the best documentation around!

What about other Unix systems?

Mostly, these days, this means a BSD variant. Yes Solaris 11 is still out there…

But if you’re asking me a question about what distro to use then you’re probably not the right audience for these alternatives. They’re good operating systems and have some real use cases that they’re possibly the best for. But for a general purpose server or desktop with the most support and compatibility? I’d stick with a Linux distro.

Summary

Yeah, this is a messy area. And it’s got messier. Everyone has their own opinion and can argue it with religious fervor. Every distro has pros and cons, and there’s no one size fits all.

For a conservative desktop or server, I’d recommend Debian. For learning marketable skills then Rocky Linux. For more cutting edge desktops or gaming… I dunno.

And then there’s SteamOS…

Should I homelab or outsource

Wed, 01 Apr 2026 12:22:01 -0400

Over on Reddit someone commented that self-hosting solutions (eg email, calendars, file sync) was too much like hard work. Needing to update TLS certificates, keeping code up to date, acting like IT support (especially if what you’re doing is used by friends and family) was becoming too much like a job.

And you know what, they could be right. Other people starting out on their homelab and self-hosted path need to be aware of this.

Firstly, it’s very important to recognise that this isn’t for everyone. If you don’t get a sense of fun or enjoyment out of doing something then why do it? “Because Google Bad” and owning your own data isn’t really enough to make someone do stuff they don’t enjoy.

Now I’m about to say something that might sound nasty, but I really don’t mean it that way. The problem could be a skills issue. The good news is that these skills can be gained, and it’s not that hard. You just have to persevere.

Let me explain.

As the original poster noticed, running a homelab is harder than just running a gaming PC and off loading everything else to Google/Microsoft/Apple. It requires “a particular set of skills” (to quote a famous movie).

I guess I started homelab type stuff in the early 90s. The tools we have today didn’t exist back then so I would create my own scripts and automations. It really helped that I found this fun and interesting; “ooh, I learned something! This is cool!”

The homelab I have today is far superior (and more complicated) than what I had then (we didn’t even have virtual servers so my spare bedroom had half a dozen machines, including two Sun SPARCs, all running and doing stuff!). Today “maintenance” of my homelab is maybe 5 minutes each morning to check for any errors and 10 minutes every 80 days to renew certs (that “complicated” bit means I haven’t 100% automated TLS cert renewal; instead I have to run one command and check the output; I really should fix that).

But this didn’t happen in 1 day; it’s taken years to get to this point. Heck, just last month I reworked by backup reporting system. I have 17 OS instances running permanently (some physical, some virtual, some cloud hosted) and they all backup and send me reports. I was getting annoyed with checking 17 reports daily. So I wrote some code and now I get 1 report that basically says “17 servers have backup reports (17 good, 0 bad)” (or alerts if something isn’t good). It’s only saved a couple of minutes per day, but it’s a lot less annoyance :-)

And this can be part of the problem, when starting out; you’re seeing enthusiasts like me who have been doing this for years and find it easy; we can do it in our sleep. You’re comparing what you’re going through and learning to something that’s been tweaked and optimised over a period of time.

And I’ll let you into a little secret; I may run my own mail server, web server, DNS, DHCP, home assistant, firewall… I have written automations to extend Alexa, to report and alert on stuff. I’ve automated server builds so I can create a new VM (with my preferred customisations) in a few minutes just with a single command. And despite all that I still use Microsoft OneDrive as a way to sync photos from my phone to my PC. Because it’s easier.

Homelabs are first and foremost a hobby. If you’re not enjoying it and don’t think you ever will enjoy it then don’t do it! But if you think it’ll be fun and you’re just frustrated now ‘cos it’s hard then maybe stick with it; you’ll learn the skills and it will get easier.

Why I'm not a fan of AI everywhere

Sat, 24 Jan 2026 21:09:14 -0500

These days it seems that AI is everywhere. It’s being pushed in your face, every application has an AI component. Even your OS is going to be agentic

In many cases I’m not a fan. But I can see some benefits under some circumstances.

Because of how many areas AI is getting into, this has become something of a long post (I’ve been putting it off for a while ‘cos I expected this!). And I’ve only covered a fraction of the potential use cases.

Background

When I’m talking about “AI” in this post then I’m mostly talking about the modern “Large Language Model” style of systems. These are, inherently, prediction systems. Based on input and training models and constraints on the output the system will generate responses that are statistically likely to be relevant.

John Scalzi, a pretty successful SciFi author, has written how xAI’s grokipedia makes shit up using himself as the subject of a query. He’s done this with other AIs and got similarly wrong results.

This, for example, is why AI’s have trouble with basic maths or why they “hallucinate”.

Despite these failures, somehow these AIs can pass bar association exams or do well in maths tests. Yet they fail to be accurate in the real world, famously inventing cases. I suspect this is because exam questions and answers are more generally known, and so show up in training data.

With this background I’m going to look at a few use cases and give my opinion on the suitability (or not) of AI.

I’m not going to discuss whether the current generation of AI is true AI (AGI), or whether the current path is going to lead to it. I’m not going to discuss how marketing has caused a terminology shift. I’m not going to discuss the potential AI economic bubble, nor the energy and water used. Nor the data and intellectual property stolen to create the current models.

What I’m going to talk about is the use of this AI.

Agentic OS

I’ll start here since I mentioned it earlier. And I really really don’t want this.

The problem, in my mind, is one of determinism. We have spent decades making computers do exactly what we tell them. We’ve invented specialized languages to try and be as precise as possible in making these statements.

Natural language is inherently imprecise. We see this, all the time, in online conversations. Emoticons (and later emojis) appeared in online text communities (eg Usenet) to try and avoid the problems caused by language ambiguities and through the lack of secondary information channels (tone of voice, facial expressions, body language); I might call my friend an arsehole or wanker as a mild humourous jab, but then use the same words to denigrate a stranger. An outsider, reading just the text, can’t tell.

So if two humans can’t understand each other perfectly (even worse if one is English and the other is American), can we expect some AI to understand my intent? I really don’t want my computer to do unexpected things because it interpreted my natural language commands differently to what I intended.

A common counter to this would be a “Human In The Loop” (HITL) process, but this really won’t work. Imagine this scenario…

Computer, open the last document I was working on
Do you want to open pornographic_document.jpg?
Dammit, no! That’s not what I meant! Go away, I’ll do it using keyboard and mouse

It would get even worse as your PC starts to control more of your life; for example if you have a smart house you won’t want to be prompted every time you ask it to turn a light on

Computer, turn on the light
Do you want me to turn on the light?
Oh STFU, and just do it without asking me in the future.

As an Alexa user since 2016 I would have smashed the thing with a hammer if it had kept asking me for verification.

We will tune the algorithm (some things don’t need HITL, others will) but there will always be edge cases.

So, no; I don’t want my computer guessing my intent. I don’t want an agentic OS. I want my interactions with my OS to be deterministic, not probabilistic.

Speech recognition

But since I mentioned Alexa, something AI should be good at is speech recognition. Because it’s predictive it should be able to better recognise speech and perform a more accurate speech-to-text conversion. Words it doesn’t clearly recognise can be statistically determined based on what it did recognise. And, similarly, to help detect false positive activations.

This can definitely assist in dictation scenarios, or even when talking to “smart devices”. It should also help with IVR systems.

I distinguish this from the “agentic”, above, because we’re not trying to determine the intent of the speech, just recognise the speech itself.

Vibe coding

Agh, OMG, no. I hate this. This is the worst of everything I don’t want about an agentic OS, made 10x worse.

Programming languages exist as a means of being explicit to the computer about what we want. Yes, it’s a skill; not just one of learning the language but also of one around how to think in a logical manner, of breaking problems down. The coding aspect of programming is, really, easy when compared to the brain power of understanding the problem and decomposing it.

It’s this latter aspect that vibe coding is replacing, and why I feel it’s ultimately a bad idea.

With vibe coding you describe the end state and, magically, the AI produces code for you. Except it doesn’t quite work so you refine the description and it changes the code. And you iterate on this dozens of times and eventually you get something you think does what you intended. Success, right? You haven’t had to learn a programming language, you haven’t had to break the problem down. You just used natural language to do stuff.

Well, it depends.

If this is for a home personal project and you’re not concerned about security or efficiency or even 100% accuracy then this is fine. Linus Torvalds has famously vibe coded and people point to this and say “Hey, if the inventor of Linux does it then it must be OK”. But this was for a personal project.

Linus has also said “I’m not a programmer anymore”. Indeed, if you read that article he “vibe codes” in real life; writes an email stating intent (pseudo code). There’s a HITL process; the person at the other end of the email, and the maintainers who control the merge process.

My small attempt at vibe coding resulted in working code… but not good code. There were architectural issues, even in the small program I wanted to write.

And there’s a big difference between “personal use” and “enterprise use”. Code you write today will be modified tomorrow (for some value of “tomorrow”). Your vibed code in your repo isn’t going to have any state. Someone, or something, is going to have to understand what was written, why it does what it does, how it works. That person might be you (I joke “I try to write good code and document it because the next person to maintain it could just easily be me”).

There’s a big difference between “working code” and “maintainable code”.

I’ve seen some people just say “Eh, the AI of tomorrow will be good enough to understand the code and build the context”, but that’s just magical thinking.

The other thing I’ve seen is “this is what code reviews are for”; this is the HITL part of the process. But this is, essentially, just moving the coding cost from the original coder to the reviewer. If a review now takes twice as long because the supplied code is harder to understand then we’re not actually winning. The original developer may code faster (although this may be a misperception) but the whole workflow could be slower.

In the past we’ve complained about “Stack Overflow programmers” who just blindly cut’n’paste code. I suspect we’ll also start seeing the same about vibe programmers.

Especially since the AI was trained on Stack Overflow (and has probably killed it, based on the traffic the site is now getting) and so has all of the bugs and security issues that will come as a result.

So, sure, vibe code for personal projects. But don’t take it anywhere near the enterprise. It will cost you in the long term.

Coding assistance

Now here I’m distinguishing coding assistance from actual coding itself. Indeed, almost the opposite; instead of telling the developer what to do we tell them what not to do.

We could have an agent sitting in the IDE that’s predicting mistakes and highlighting them. There are already systems that deal with this, but I could see an LLM based AI being better at detecting bad coding patterns.

Today an LLM takes too many resources to be efficient locally (umm, unless we give every developer a beefy GPU in their laptop?) so this may be better as part of a SaaS scan, but smaller models may be more efficient in the future.

Having a model that can predict errors and warn the developer while they’re coding can cut down the detect/fix cycle time, improve developer efficiency and reduce overall development time. The HITL process is immediate.

Meeting summaries

I’ve worked with people who said “wow, this is great; it’s saved me so much time”. But whenever I’ve tried to use it I’ve found it… lacking. The problem, for me, is that it loses nuance and can easily overlook an essential statement. When you try to summarise an hour long contentious meeting getting a summary of “we discussed foo; bar was mention; baz was suggested as a solution” is just not good enough (although AI will expand that out to 5 or 6 paragraphs ‘cos it loves to be verbose). I’ve not seen it be wrong but I’ve seen it miss important stuff.

Definitely a scenario with HITL is essential. Don’t rely on AI, here.

Document/website/whatever generation

I hate this. I mentioned, in the intro, how AI has caused lawyers issues because the produced document referenced non-existent cases.

I’ve seen marketing websites and “apps” (whatever was meant by that) that are AI generated, and came away wondering “just WTH did any of that mean”.

I’ve seen proposals that are so generic and meaningless that I’ve laughed at them and told the vendor to go away (to be fair, I’ve also told vendors with lazy sales people to do the same thing).

But the thing that gets me about a lot of AI generated content is the banality of a lot of it. Now maybe I’m just being exposed to the worst of the worst (eg on LinkedIn) but there seems to be a lack of tone to the document. It’s just bland and lacking anything to keep me reading.

I’ve been told by people I know personally that when they read this blog they hear my voice in their head; I write this as I would speak it. AI generated content seems to lack this. And that leads to a form of uncanny valley. I get a gut aversion to the content.

Can I tell AI content from human content all the time? Heck, no. I probably get it wrong more often than not. But that just means other content is just as bad :-)

Using an AI to create bad content is not excused because humans also create bad content.

(And, hush; no calling this blog “bad content”!).

Marketing emails

Just FOAD.

I hate marketing spam at the best of times; sending AI generated slop is a quick way of getting me to hate you and your company and to tell everyone to avoid you.

In my career I’ve had people cold-email me; I’ve had these cold-emails lie to me (“I’ve tried to contact you…”; no you haven’t, I’d have blocked your company if you had!). I really don’t want AI spam in my inbox.

Fortunately I’ve retired, but this still triggers me :-)

As a joke (which I formulated over a decade ago), I’ve wondered if spam is going to be the path to true AI. See, the spammers use better and better systems to generate email that looks like it’s from a human. Which means we need better and better solutions. And this is what is ultimately going to lead to a true AI… who might just destroy the world because of all the content it learned from!

Image generation

Y’know; I’m of two minds on this one. For an idiot like me I can see the use of image generation to help get messages across. You might notice a distinct lack of pictures on this blog; I’m not a visual person and my attempts at drawing might be considered “scribbles” on a good day. So a “vibe coded” image might be beneficial? Maybe?

This is kinda like vibe coding in general; for personal use I could see a benefit, but be very aware of using it in an enterprise or commercial situation. And doubly so if using it for marketing; you don’t want a potential customer to laugh at you because your images are so obviously bad.

And here’s another minefield. I said I wasn’t going to get into stuff like IP theft and the like. But this is an area that is rife for abuse. I can’t stay silent with things like Grok can create Child Sexual Abuse Imagery(CSAM).

Now it’s not just Grok that can do this, but its the one with the worst guard rails.

Guard rails

And this kinda takes me all the way back to the beginning; when we’re parsing the intent of human language how can we prevent abuse. Humans are already bad at this. Con men are adept at bypassing guard rails. Business Email Compromise is based on the precept that people can be convinced to bypass guard rails.

We call it “prompt injection” but it’s really the same thing; we convince the AI to do things it’s not meant to do. Whether it is creating porn images of celebrities, or CSAM, or revenge porn, or…

We’re already seeing use of this in scam calls.

In the future we’re going to see a LOT of fake images during election season. Previously Trump used them to generate fake images promoting himself; eg hanging out with people of colour. In the future we’re going to see fake videos of political opponents doing/saying things they never did.

How can we build the necessary guard rails, especially since this stuff is open source and can be run locally? Fact checkers won’t help because we already have people believing “alternate facts”. Consensus reality appears to be a thing of the past, and everyone lives in their own private reality.

(This is depressing. Sorry! And it’s not really an AI problem, but it’s something AI will make worse).

I think we’ve entered a future where you can’t believe anything you don’t see or here, in person, with your own eyes and ears. And, as noted, even that’s not infallible!

Summary

I’m not, in general, a fan of AI. I don’t want it on my machine, I don’t want it writing my code. I definitely don’t want it writing code for my employer (if I still had one). I’ve modified my web browser configs so that my search excludes AI results, because they’re frequently wrong.

But I can see areas where, as a consumer of services it can make my life better (eg IVR systems).

Wherever this form of AI is used, it’s essential that HITL is present. And you need to evaluate the cost of that; it might turn out that you’ve shifted cost from stage 1 over to stage 3 of a process (in programming terms you’ve shifted right!) and potentially increased overall costs as a result.

I suspect this whole post will be controversial simply because of the areas I’m covering. Everyone has their own opinions. But that’s what this blog is here for; for me to opine on things!

If you have an AI area you want to hear me ramble on about, let me know :-)

And this post was written without the help of any AI; indeed I wrote it using vi in an ssh window.

TLS Settings in 2025

Tue, 02 Dec 2025 13:10:49 -0500

Back in 2016 and then in 2020 I described how to get an A+ score with your TLS config.

That was 5 years ago. Since then OpenSSL and Apache have both advanced and there’s even more options than before. Nicely OpenSSL can now also use the official TLS names for ciphers, so we don’t need to keep switching between entries reported by Qualys SSL Labs and the OpenSSL internal names.

I’m also migrating to Debian, which has some differences.

Since we’re at it, we can also set security headers and make sure we have a proper CAA DNS entry set.

Support RSA and ECDSA certificates

In 2021 I described how to configure apache to support modern certificates based on elliptical curves, as well as traditional RSA certificates. That blog entry also describes how I use Dehydrated to get these certs from LetsEncrypt. With newer versions of Apache (eg RedHat 8 onwards, or Debian) we can specify this pretty easily; e.g.

	SSLCertificateFile /etc/apache2/SSL/sweharris/fullchain.pem
	SSLCertificateKeyFile /etc/apache2/SSL/sweharris/privkey.pem

	SSLCertificateFile /etc/apache2/SSL/sweharris_ecdsa/fullchain.pem
	SSLCertificateKeyFile /etc/apache2/SSL/sweharris_ecdsa/privkey.pem

Debian Ciphers Out Of The Box

The default configuration for Debian is SSLCipherSuite HIGH:!aNULL. You might think that’s good. Unfortunately it’s not. Reading the openssl-ciphers(1) manual page we see

HIGH - “High” encryption cipher suites. This currently means those with key lengths larger than 128 bits, and some cipher suites with 128-bit keys.

This setting results in a B grade by SSL Labs because a number of those ciphers do not include “forward secrecy”. Forward secrecy (FS) is very much a nice thing to have; it means that if your private key ever leaks then it can not be used to decrypt old saved traffic logs (which is what “harvest now, decrypt later” type attacks try to do; save your data now, decrypt it in the future). After all, key management is one of the hardest parts of encryption!

So what OpenSSL means by “HIGH” is simply the bit strength.

Debian also defaults to a complicated protocol list; SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1. The practical result of this is versions 1.2 and 1.3 are supported. I’m guessing it’s written this way so that if a mythical 1.4 was ever supported then this line wouldn’t need changing.

Remove all the non-FS ciphers

Fortunately the SSL Labs results page can be used to get a list of all the ciphers that were supported by the default configuration and it flags those that provide FS. A bit of cut’n’paste and editing resulted in this list:

	TLS_AES_128_GCM_SHA256
	TLS_AES_256_GCM_SHA384
	TLS_CHACHA20_POLY1305_SHA256
	TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
	TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
	TLS_DHE_RSA_WITH_ARIA_128_GCM_SHA256
	TLS_ECDHE_ECDSA_WITH_ARIA_128_GCM_SHA256
	TLS_ECDHE_RSA_WITH_ARIA_128_GCM_SHA256
	TLS_DHE_RSA_WITH_AES_128_CCM
	TLS_ECDHE_ECDSA_WITH_AES_128_CCM
	TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
	TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
	TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	TLS_DHE_RSA_WITH_ARIA_256_GCM_SHA384
	TLS_ECDHE_ECDSA_WITH_ARIA_256_GCM_SHA384
	TLS_ECDHE_RSA_WITH_ARIA_256_GCM_SHA384
	TLS_DHE_RSA_WITH_AES_256_CCM
	TLS_ECDHE_ECDSA_WITH_AES_256_CCM
	TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
	TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
	TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256

Unfortunately it doesn’t seem as if OpenSSL has a short-cut for these “FS” ciphers, so we have to list each one separately on the SSLCipherSuite line.

We need CBC as well

As in 2020, this still results in some older software being unable to negotiate a cipher, even though they support TLS1.2. SSL Labs still flags CBC ciphers as weak, but it doesn’t reduce the score. So let’s add these two in:

	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

Is this too much?

What I noticed in the results was that although we offer every cipher, all the simulations provided by the SSL Labs tester ended up only using a subset of them! So we could provide a small list of ciphers

	TLS_AES_128_GCM_SHA256
	TLS_AES_256_GCM_SHA384
	TLS_CHACHA20_POLY1305_SHA256
	TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
	TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
	TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
	TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
	TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
	TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

This is shorter and so may (especially on TLS1.2) result in smaller packets. The downside is that an odd client may need one of the other ones.

But with this configuration, we get the following results:

	SSLCipherSuite TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

HSTS and other headers

To round this off we need to specify strict transport security headers.

	Header set Strict-Transport-Security "max-age=31536000; includeSubDomains"

While we are here we can also add some other good and useful security headers.

I consider these to be a good default, but you might have different needs.

	ServerSignature off
	ServerTokens Prod

	Header set X-Content-Type-Options "nosniff"
	Header set X-XSS-Protection "1; mode=block"
	Header set X-Frame-Options "SAMEORIGIN"
	Header set Referrer-Policy "no-referrer"
	Header set Permissions-Policy: interest-cohort=()
	Header set Content-Security-Policy: script-src 'self'

The Content-Security-Policy entry is very likely to need to be customised for your site; e.g. if you include fonts from Google you’d need to include those here. Check out Scott Helme’s description for an introduction to CSP.

And a Security Headers scan nicely returns an “A” score with that setup.

Restricting what CAs can be used

A well behaved Certificate Authority will check to see if the requested domain has a Certificate Authority Authorization (CAA) policy defined and, if one is found, will only issue certificates if permitted. This is set in DNS. Since I use LetsEncrypt I have a simple policy:

	@       IN      CAA     0       issue   "letsencrypt.org"

HTTP/2 support

This isn’t really needed but it might be useful to set up. It can speed up traffic and reduce overhead with TLS1.2. With Debian it’s as simple as doing a a2enmod http2. On RedHat systems you may need to dnf install mod_http2 first.

The result

And all the clients that support TLS1.2 or TLS1.3 and are simulated by SSL Labs connect cleanly.

A personal growth moment

Sun, 26 Oct 2025 20:39:25 -0400

The recent AWS outage led to a slew of It was DNS jokes.

It reminded me of a time maybe 12 or 13 years back. I was part of the architecture/engineering team for the company’s Unix authentication product. Basically every login, su, privilege escalation call went through our code.

So when things went wrong we got the blame.

One day a call got escalated to me. It went something like:

Caller (C): It’s taking a long time to login
Me (M): How long?
C: About 10 seconds
M: (thinking “hmm, that sounds like a DNS timeout”). Does it happen if you login again?
C: No, just on first login
M: It’s DNS
C: It can’t be, otherwise we’d have got an alert
M: OK, so what happens is that the SSH daemon is trying to do a reverse DNS lookup on the IP address connecting to it. If DNS isn’t working properly then this can take some time. But the nscd process caches this for a while, which is why the second login is fast.
C: What can we do about it?
M: Well, we can change the sshd_config file to disable those lookups but it might not always work

Then we went through the process of checking their DNS configs were correct, being able to demonstrate an issue using nslookup, and then configuring SSH to disabling the lookup. The caller went away happy; their logins worked fast again. I told my team of a potential DNS issue.

2 hours later a global email announcement of DNS issues was made.

Yeah, it was DNS.

How I’ve grown

But going through this again in my mind, I realised where I didn’t go far enough.

Yes, I solved the caller’s problem. People were happy that my team was responsive. The business kept running. I did my job :-)

But what I didn’t do was to take this to the next step. Was it worth disabling this configuration globally? It would be any easy change to make and to deploy a new rpm which we could push out and it would deploy as part of the patching process.

What I should have done was to discuss this with the Linux and Unix engineering teams to come up with a consensus of whether to leave the default values as-is, or to disable this lookup. I think it would have made an interesting discussion, especially considering the potential impact. It likely would have had the Linux/Unix teams trying to evaluate the likelihood of that impact (perhaps collecting the config files from the 60,000+ endpoints).

Clearly somewhere in the past decade I learned to look further than just “incident response” and into a “what could be done to prevent this from happening again” mindset.

Because it’s always DNS.

Don't ask 'manpage' questions in interviews

Thu, 23 Oct 2025 12:06:39 -0400

One of the things you might see a lot on sites like LinkedIn is the recommendation to not ask interview questions for stuff that is easy and trivial to look up. For example, asking about command line options to a program. To an extent I agree with this. I always need to look up the “key field” options to the sort command (especially since they changed from when I learned the command!).

Indeed, when I interviewed for a job in 1999 one of the questions was “How would you count the number of times each shell was used in a NIS password map?“. Off the top of my head I said

ypcat passwd | cut -d: -f7 | sort | uniq -c | sort -nr

Well, what I really said was “ypcat password filtered through cut minus d 7 (umm, I think it’s field 7; I’d need to check the manpage) filtered through sort filtered through uniq minus c (shrugged shoulders) filtered through sort minus nr (to get them ordered by use)“.

And that’s the thing; I knew the commands and how to put them together; but a trivial detail like the right field of the password file is easy to look up (man 7 passwd) that it’s not really important.

When a joke taught me something

Back in the long distant past I remember a joke going around Usenet that the ls command had so many options to it that it was harder to find a letter that wasn’t an option. So I added a joke question to the list of questions I asked people interviewing for a senior security engineer role. This was for someone who was going to build code that would run on 60,000+ different machines across multiple different operating systems (primarily RedHat, Solaris). I’d already dug deep into their knowledge of PAM and NSS and SSH, so I thought to lighten things with an easy joke question:

This question is a bit of a joke; there’s so many options to the ls command, can you list 10 of them and what they mean? I don’t expect you to get 10 and don’t worry if you can’t; it’s just a joke question.

Now this breaks the “don’t ask manpage questions” guidance.

I thought it would be easy. After all I use 4 options (-lart) daily; easy to remember for anyone who had spent time in the Scary Devil Monastery (SDM) on Usenet. And since the early ‘90s I had alias ls='/bin/ls -F' in my profile. So coming up with 5 more would be easy.

I just went about it logically.

We start with

-l - long listing
which leads to -g for group information (bonus points for mentioning the historical difference between BSD and SysV implementations of -g)
also leads to -L for following symlinks
-a - all files
which leads to -A for all files except . and ..
-r - reverse sort
Which leads to -R for recursive lists
-t - to sort by modified time
which leads to -c for metadata change time
and to -u for access time
and the -c takes us to -C for forcing column based output (e.g. in pipelines)
which takes us to -1 for single column output
Inhabitants of SDM might also have heard of LARTd (the LART daemon) so we have -d to show the directory and not the contents of the directory
And I use -F to add a filetype marker (e.g. * for executable, @ for symlink) to the output

So, yeah, I could get 14 without really thinking, just by following a chain through a starting point. There are other options I’ve used in the past, but that’d take some thinking and, yes, checking the manpage to remember them :-)

So, sure, I thought 10 would be an achievable goal.

What I learned

Only 1 person was able to get 10. That surprised me.

One person mentioned an option I didn’t know (from GNU ls). Another used GNU long options (which was fair; I hadn’t specified traditional options).

What surprised me most was that some people didn’t understand that -al wasn’t an option, but was two options (-a and -l combined). That’s almost a basic convention of Unix; how can you go for a senior role without knowing that basic knowledge?

And what disappointed me was that no one took a logical approach; they just tried to remember stuff they’d used in the past. Where I took a “l leads to L; t leads to c leads to C leads to 1” approach, they just struggled.

When I mentioned some options to people (e.g. -A) and asked them what it did they didn’t know. Which, to me, feels like they didn’t know the functionality of the command and what it could do. This could have led to inefficient code or poor scripts where they had to work around problems that didn’t have to have existed.

Summary

It turned out that my joke question caused interviewees some stress, even when I emphasized the joke nature of it. But I did learn a lot from it!

Hands off Rocky/Debian installs

Wed, 13 Aug 2025 18:35:34 -0400

This is a long long post because I’m providing a lot of configuration files and explanation. I could have split this into multiple posts, but I felt it made sense to put it all in one entry

In my homelab I use virsh to manage QEMU/KVM virtual machines. I might want to spin up a test VM (oh, say Debian 12) to do some playing around and then destroy it again. And, naturally, I don’t want to do an interactive install each time; I just want to run a script and 5 minutes later have a shiny new VM to play with. And I don’t want a GUI; these are “server” builds, and so should have a serial console.

Fortunately both Rocky Linux (via RedHat kickstart) and Debian Linux (via preseed) has the ability to perform hands-off installations.

Pre-requisites.

Pretty much all you need is a local webserver that can server out the kickstart/preseed file, and any other file you want.

For historical reasons (I started doing this with RedHat, then later CentOS) the base of my tree is at http://10.0.0.137/CentOS/kickstart. I still use this base for my Debian builds, even though it isn’t needed.

A fast internet connection is useful. You could mirror upstream installer trees (I used to do this), and then install locally, but now I have gigabit internet access the speed of the download isn’t so much the limiting factor. As part of my migration to Debian, I installed apt-cacher-ng on a machine so repeated installs won’t hit the upstream infrastructure so much (packages are served locally instead) and this ended up only saving 4 or 5 seconds. (Enterprise’s probably have a local mirror already because you shouldn’t be allowing direct access to external sites!)

Scripting

Now I’ve scripted everything I’m going to describe below so I don’t actually run any of these commands. But I’m going to explain the low level commands so you can see what to do and can write your own wrappers. My scripts aren’t that useful because they are very dependent on other parts of my highly unique setup (e.g. it looks at the inventory file, if it sees the hostname I want to build in there then it extracts the MAC address and allocates this to the VM; in that way the VM will get a consistent value from DHCP ‘cos the DHCP server config is also built from the same file!)

Allocating disk space

KVM can use files on the filesystem or volumes. For example, to create a 10G logical volume I could run

   sudo lvcreate --wipesignatures n -L 10G -n vm.debian13 SSD

This will create /dev/SSD/vm.debian13

(I have a naming scheme; all VM logical volumes are called vm. followed by the name of the VM. That way I can do an lvs and know exactly what each volume is for).

Now when running virt-install to create the VM you would pass this paramter

   --disk path=/dev/SSD/vm.debian13`

Alternatively if you want to use a QCOW image on the filesystem would would specify the path to the image and the size you want; the paramter then becomes something like

   --disk path=/var/lib/libvirt/images/debian13.img,size=10G

Networking

I have 3 VLANs that are visible on the host as br-lan, br-guest, and br-iot. Normally I build my VMs on the main LAN, but I could build on any of the VLANs.

This is specified with the virt-install network paramter;

   --network=bridge=br-lan,target=v-debian13

We can also specify a MAC address here if we wanted to:

   --network=bridge=br-lan,target=v-debian13,mac=01:02:03:04:05:06

The target value will name the virtual interface so, once again, I can quickly see what running VMs are on what bridges:

   % brctl show
   bridge name     bridge id               STP enabled     interfaces
   br-guest        8000.b0416f0e52ab       no              enp1s0.11
                                                           v-pinky9-guest
   br-lan          8000.b0416f0e52ab       no              enp1s0.10
                                                           v-debian13
                                                           v-pinky9
                                                           v-test-debian
   br-iot          8000.b0416f0e52ab       no              enp1s0.12
                                                           v-pinky9-iot

Or by looking at the ip a output we can also see it; eg

   19: v-debian13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UNKNOWN group default qlen 1000
       link/ether fe:54:00:92:38:7f brd ff:ff:ff:ff:ff:ff
       inet6 fe80::fc54:ff:fe92:387f/64 scope link
          valid_lft forever preferred_lft forever

Other common parameters

To name the VM, -n debian13

To prevent virt-install from allocating a graphics console we specify `--nographics

To allocate RAM, we specify --ram 1024 (although the amount allocated may vary across the different OS releases; we’ll get to that, below).

To specify the number of virtual CPUs, --vcpus=1

Installing different OSes

Now we’ve got the basics out of the way we can start to look at the differences between different OS builds. For ease of reading when it comes to the network and disk allocations, I’ll just ... the value since I’ve described them in detail above.

Rocky Linux

I have configurations for Rocky 8 and Rocky 9. Most of the parameters to virt-install are the same. The differences are:

Rocky 8 variables

   ks=rocky8
   repo=https://dl.rockylinux.org/pub/rocky/8/BaseOS/x86_64/kickstart/ 
   ram=3073 
   os=rocky8

Rocky 9 variables

   ks=rocky9 
   repo=https://dl.rockylinux.org/pub/rocky/9/BaseOS/x86_64/kickstart/ 
   ram=4096 
   os=rocky9

Kicking off the Rocky install

We can then run the install process

   virt-install \
       --noreboot \
       --vcpus=1 \
       --nographics \
       --machine pc-i440fx-rhel7.6.0 \
       --accelerate \
       -v \
       -n $name \
       --os-variant=$os \
       -r $ram \
       --network=... \
       --disk ... \
       -l $repo \
       -x "inst.ks=http://10.0.0.137/CentOS/kickstart/$ks.cfg
       ksdevice=ens2 ip=dhcp console=ttyS0,9600"

(that last -x line entry is really on one line; I’ve split it here for readability)

The ksdevice entry has changed over time (eg on CentOS 7 it was eth0) but it’s been consistent between Rocky 9 and Rocky 9.

The --machine entry is maybe not needed any more and might be a holdover from older host OSes (I’m now using Rocky 8 and Rocky 9 as the hypervisor OS). Similarly the --accelerate option may no longer be needed!

The -l flag is where virt-install will try to download the installer kernel/ramdisk from. We’re using the files in the kickstart tree.

The -x flag specifies what is passed to the kernel as parameters. In this case we specify where to find the kickstart configuration, as well as the network device, how to get an IP address and to use a serial console.

Now in this installer I specify --noreboot so that after the VM has been built it’s shutdown. This is so I can do some post-install VM tuning; specifically reduce the memory configured. Rocky Linux will run in a lot less memory than it needs to install (eg 512), so after the install has completed I reset the memory requirements and restart the VM

   virsh setmaxmem $machine 512M --config
   virsh setmem $machine 512M --config
   virsh start $machine

The real magic starts in the kickstart file!

Rocky 8 kickstart config

We’ll break this down into parts

   install
   url --url https://dl.rockylinux.org/pub/rocky/8/BaseOS/x86_64/os
   poweroff

   lang en_US.UTF-8
   keyboard us

   network --onboot yes --device eth0 --bootproto dhcp --ipv6 auto

   rootpw  --iscrypted $6$...
   authconfig --enableshadow --passalgo=sha512

   firewall --disabled
   selinux --disabled

   timezone --utc America/New_York

So far this is pretty understandable. Note the rootpw entry is pre-encrypted (you could just that this from an existing /etc/shadow entry, for example) so there’s no plain text secrets.

I also disable firewalld and SELinux (yeah yeah, security guy disabling security…)

Now we get to creating the partitions; I create a 500M /boot, a small swap partition and set the rest to be an ext4. You can define whatever layout you want here; it’s pretty simple!

   zerombr
   clearpart --all --initlabel
   part /boot --fstype=ext4 --asprimary --size=500
   part swap --asprimary --size=512
   part / --fstype=ext4 --asprimary --grow --size=1
   bootloader --location=mbr --driveorder=vda --append=" crashkernel=auto quiet" --timeout=0

Next we define what packages we want. I typically want to install as small as possible, with some specific extras. I also remove unnecessary firmware

   repo --name=AppStream --mirrorlist https://mirrors.rockylinux.org/mirrorlist?repo=AppStream-8&arch=x86_64

   %packages
   @^minimal-environment
   python36
   wget
   ksh
   dos2unix
   logwatch
   tar
   postfix
   bind-utils
   bc
   -kdump
   -iwl100-firmware
   -iwl1000-firmware
   -iwl105-firmware
   -iwl135-firmware
   -iwl2000-firmware
   -iwl2030-firmware
   -iwl3160-firmware
   -iwl3945-firmware
   -iwl4965-firmware
   -iwl5000-firmware
   -iwl5150-firmware
   -iwl6000-firmware
   -iwl6000g2a-firmware
   -iwl6050-firmware
   -iwl7260-firmware
   %end

Since I removed kdump, we can also tell this to be disabled

   %addon com_redhat_kdump --disable --reserve-mb='auto'
   %end

And then some post-install steps. I’ll get to what the “post.tar” file is later in this post because I use a consistent method for Rocky and Debian

   %post
   exec 1>/root/ks-post.log 2>&1
   tail -f /root/ks-post.log > /dev/console &

   alternatives --set python /usr/bin/python3

   mkdir /tmp/post
   cd /tmp/post
   wget -q http://10.0.0.137/CentOS/kickstart/post_r8.tar
   tar xf post_r8.tar
   sh runme
   cd /tmp
   rm -rf post
   %end

And that’s it! The complete file is available here

Rocky 9 kickstart config

This is kinda the same deal, but with some minor differences. Rather than detail the whole file again, my config is available here

Mostly the differences are in url and repo entries, as well as allocating more space to boot and swap.

Debian Linux

The Debian installer works in a different way to Redhat’s. It exposes things in a more “raw” level; essentially each part of the installer has a tag for the questions it asks and you can “preseed” answers by defining the tag type and value. It’s also a little inconsistent around when it reads the file, so some of the values need to be passed to the kernel for them to be read.

There are fewer variables that need to be considered for virt-install; basically just the OS version and codename

Debian 12 variables

   version=12
   codename=bookworm

Debian 13 variables

   version=13
   codename=trixie

Kicking off the Debian install

This looks very similar to the Rocky one, but the “-x” options are a lot different

   virt-install \
     --noreboot \
     --vcpus=1 \
     --nographics \
     --accelerate \
     -v \
     -n $name \
     --os-variant=debian12 \
     --ram 1024 \
     --network=... \
     --disk ... \
     -l http://ftp.us.debian.org/debian/dists/$codename/main/installer-amd64/ \
     -x "console=tty0 console=ttyS0 auto
     url=http://10.0.0.137/CentOS/kickstart/debian$version.cfg
     language=en country=US locale=en_US.UTF-8 keymap=us
     netcfg/get_hostname=debian netcfg/get_domain=spuddy.org"

(that last -x line entry is really on one line; I’ve split it here for readability)

The “preseed” file is specified by the auto url=... option. But we also need to specify other stuff (language, country, locale, keymap) because the installer tries to use those values before the preseed file is read. Oddly enough we also need to specify the hostname and domain name here as well, even though we get real values from DHCP.

Debian 12 preseed config

This is a much more hard-to-build file than the kickstart one.

Lines beginning d-i mean “debian-installer”. There are also tasksel and popularity-contest lines. And because I’m installing postfix we also need some postfix lines. This took lots of trial and error and reading docs and examples (and help from r/debian).

   #_preseed_V1

   # These values all based on
   #   https://d-i.debian.org/manual/en.amd64/apbs04.html
   #   https://michael.kjorling.se/pages/debian-12-bookworm-preseed/preseed.cfg

   d-i debian-installer/language string en
   d-i debian-installer/country string US
   d-i debian-installer/locale string en_US.UTF-8
   d-i keyboard-configuration/xkb-keymap select us

Those lines duplicate the options we passed on the kernel parameters; they may not be needed!

   # Where to find the installer
   d-i mirror/protocol string http
   d-i mirror/country string manual
   d-i mirror/http/hostname string http.us.debian.org
   d-i mirror/http/directory string /debian
   d-i mirror/http/proxy string http://10.0.0.2:3142

   # Suite to install.
   d-i mirror/suite string bookworm

Here we’re defining where to install from; the proxy line is telling the installer to use the apt-cacher-ng server. We’re installing Debian 12 aka “bookworm”.

Aside: I really dislike using codenames in configs. It’s so much easier to remember “11” or “12” or “13” rather than … well, I don’t remember what 11 is; 12 is bookworm, 13 is trixie. It’s cute, but it’s bad. And, yes, I’m staring very hard at Apple…

The config continues and is pretty easy to understand

   d-i passwd/root-login boolean true
   d-i passwd/root-password-crypted password $6$...

   # To create a normal user account.
   d-i passwd/user-fullname string Stephen Harris
   d-i passwd/username string sweh
   d-i passwd/user-password-crypted password $6$...
   d-i passwd/user-uid string 500

   d-i time/zone string US/Eastern

And now we get to partitioning. Debian comes with some default templates, but they tend to put the OS at the beginning of the disk and the swap at the end. I wanted swap at the beginning so I could extend the disk and simply resize/extend the partition if I needed more space. So this will create a 1024M swap file and then the rest is a single ext4 root partition

   d-i partman-auto/method string regular
   d-i partman-auto/expert_recipe string                         \
         boot-root ::                                            \
                 1024 1024 1024 linux-swap                       \
                         $primary{ }                             \
                         method{ swap } format{ }                \
                 .                                               \
                 1 1 -1 ext4                                     \
                         $primary{ } $bootable{ }                \
                         method{ format } format{ }              \
                         use_filesystem{ } filesystem{ ext4 }    \
                         mountpoint{ / }                         \
                 .
   d-i partman/confirm_write_new_label boolean true
   d-i partman/choose_partition select finish
   d-i partman/confirm boolean true
   d-i partman/confirm_nooverwrite boolean true

We can then configure what additional repositories are in use:

   d-i apt-setup/cdrom/set-first boolean false
   d-i apt-setup/non-free-firmware boolean true
   d-i apt-setup/non-free boolean true
   d-i apt-setup/contrib boolean true
   d-i apt-setup/disable-cdrom-entries boolean true

Now we define the OS base (“standard” with “ssh-server”) along with any additional packages; note that I chose postfix as my mail server; that will also need configuring later on.

   tasksel tasksel/first multiselect standard, ssh-server

   d-i pkgsel/include string ksh sudo curl postfix rsyslog binutils dump bsd-mailx apt-file rsync collectd-core
   d-i pkgsel/upgrade select full-upgrade

   popularity-contest popularity-contest/participate boolean false

Grub information:

   d-i grub-installer/only_debian boolean true
   d-i grub-installer/bootdev  string /dev/vda

   d-i finish-install/reboot_in_progress note

Now comes the postfix config

   postfix postfix/mailname string spuddy.org
   postfix postfix/main_mailer_type string 'Internet with smarthost'
   postfix postfix/relayhost string mailhost.spuddy.org

And then a post-install command that does something similar to the %post section of kickstart. This command is run at the end of the installer.

   d-i preseed/late_command string chroot /target sh -c "cd /tmp && /usr/bin/curl -o post.tar http://10.0.0.137/CentOS/kickstart/post_d12.tar && /usr/bin/tar xf post.tar && /bin/sh -x /tmp/debian-postinstall"

The complete file is available here

Debian 13 preseed config.

This is almost identical, except the mirror/suite refers to trixie and the late_command refers to post_d13.tar. Just for completeness, it’s here.

The post-install steps

For all 4 build types there’s a post-install step. This primarily involves downloading a tarball, extracting it, and running the script that’s inside it.

What this does is deploy some standard configurations to each machine; e.g. it ensures my account has a standard $HOME (with my ssh public keys), might add an additional repos (eg EPEL for Rocky), configures postfix for my smarthost, creates email aliases so mail to my account and to root is forwarded properly and so on. Because it’s a full script you can do almost anything; it runs inside the build chroot environment.

An example of this might be my Debian 12 script:

   # Extract all the files I want
   cd /
   /usr/bin/tar xvfp /tmp/files.tar
   /bin/rm /tmp/files.tar
   
   # Remove "quiet" from kernel boot params
   sed -i 's/"quiet"/""/' /etc/default/grub
   
   # Ensure grub is updated with any changes that might have made
   /sbin/update-grub
   
   # Fix my account
   /usr/bin/chsh -s /bin/ksh sweh
   /sbin/usermod -a -G sudo sweh
   /sbin/usermod -a -G root sweh
   
   # Setup postfix correctly
   echo 'root: root@mailhost' >> /etc/aliases
   /usr/bin/newaliases
   
   /usr/sbin/postconf append_dot_mydomain=yes
   /usr/sbin/postconf mydomain=spuddy.org
   /usr/sbin/postconf 'mydestination=$myhostname, localhost.$mydomain, localhost'
   /usr/sbin/postconf 'myorigin=$myhostname'
   /usr/sbin/postconf -X myhostname
   /usr/bin/systemctl enable postfix
   
   # Remove static hostname
   echo "" > /etc/hostname
   
   # Remove this annoying entry from hosts file
   sed -i '/127.0.1.1/d' /etc/hosts
   
   # Fix PAM SSH
   sed -i 's/user_readenv=1//' /etc/pam.d/sshd
   
   echo `date`: Post script completed

The files.tar it refers to is embedded into the post_d12.tar script and contains things such as

   ./etc/apt/apt.conf.d/00aptproxy
   ./etc/sudoers.d/sweh
   ./etc/sysctl.d/00-dmesg.conf
   ./etc/default/grub.d/console.cfg
   ./etc/default/grub.d/apparmor.cfg
   ./etc/logwatch/conf/services/zz-disk_space.conf
   ./etc/ntp.conf
   ./etc/cron.daily/need_patches
   ./home/sweh/.profile
   ./home/sweh/.envfile
   ./home/sweh/.forward
   ./home/sweh/.ssh/authorized_keys
   ./root/.ssh/authorized_keys

(that’s not the complete list, just a sample)

Basically, any “standard” changes you want deployed to all my VMs is performed in this script, and it can be customised to each OS variant.

Conclusion

What I’ve described here is suitable for a homelab. But some of the same concepts can be applied to enterprises as well. For example, that kickstart file could be modified to be regional specific to pick a local replica to get the repos. The post-install script could call out to something like ansbible or puppet to do proper configuration management (even auto install and configure something like Apache web server if the machine is defined as “webserver” in an inventory system). The automated install options provided by Rocky (RedHat) and Debian are very flexible.

In my case, I can just run deb_install -13 new_host and it will create the LV for disk space, and run virt_install with all the required parameters. 5 minutes later I have a new fully patched VM to test and play with, with sane default values.

When automation is a problem

Thu, 17 Jul 2025 08:39:28 -0400

In the past I’ve pushed for automation of server builds, of application configuration.

Indeed, for my home setup, I’ve been using ansible for over a decade; I still see a config file for CentOS 6 postfix dated 2015.

I’ve started a migration from CentOS/Rocky Linux (i.e. RedHat) to Debian for my personal servers. And I’ve realised this automation is causing me more problems than it’s solving:

Differences between distributions

Debian is configured very differently to RedHat. This shows in many places, but one of the most obvious is with Apache. With RedHat the configuration files all live in /etc/httpd. With Debian it’s /etc/apache2. With Debian you manage symlinks with commands such as a2enmod and a2ensite; with RedHat it’s more “just put stuff into the config directory”.

Now you can put stuff into the Debian directory, but it’s different (/etc/apache2/sites-enabled and /etc/apache2/conf-enabled and /etc/apache2/mods-enabled, vs /etc/httpd/conf.d and /etc/httpd/conf.modules.d)

And the contents have to be different as well; eg the log directory on RedHat Apache is just logs/ but on Debian its ${APACHE_LOG_DIR}.

Apache is just one example. Other tools have similar configuration differences.

All this means that the configs I built for RedHat pretty much need to be rewritten from scratch for Debian. It was bad enough having to deal with differences between RHEL 6, 7, 8, 9 but there was sufficient consistency.

Maybe everything is a pet.

When I looked at all my playbooks I realised a large number of them were targeting single machines. Which kinda makes sense; in my home environment every OS instance is doing something different; it might be my desktop, or a media player, or a plex server, or a home assistant server, or a bastion host, or a router, or…

Pretty much only two machines were configured “the same” (or close to, allowing for different hosting providers). Everything else was custom.

My Apache config for my bastion “reverse proxy” is very different to my config for the server hosting this blog. (They’re on different OS releases, to start with!).

So although I had tried to automate the build for each of these pets, I kinda never used those playbooks again ‘cos I never rebuilt the server; if there had been a failure I would have restored from backup.

Reduced build automation

So rather than rewrite my playbooks and create either duplication (different OSes with different playbooks and config files) or horrendous conditional logic in the config files (which really makes it harder to read, and brings in tech-debt which will need to be removed as I finally move off the older OSes), I’m mostly doing things by hand.

The Debian OS deployment, itself, is automated with netinstall and preseed configurations. In this preseed I have a post-install step that deploys common configs (eg telling postfix to point to my mailhost), handles some custom scripting and so on.

But from then on, the upper layers are manual. One server got a docker deployment; another got grafana and influxdb (the configs and data for those got migrated from the old machine to the new); the reverse proxy got apache2; and so on.

Maintenance automation

There is still some requirement for automation. Most commonly to redeploy new TLS certificates every 80 days. Because most of my servers can’t be reached from the internet and because those that are accessible may not have DNS pointing to them (warm standby) I can’t just use certbot to manage them simply. So, instead, I use dehydrated to get the certs and then deploy them to all the servers and services (apache, postfix, grafana, etc etc) with a set of ansible playbooks.

Whether I keep using ansible for this “maintenance” work or if I’ll write a set of scripts that run via ssh I’m not sure.

Summary

In an enterprise environment there can be little doubt that automated processes are a win. They can allow for hands-off deployments; repeatable, testable processes; controlled access and more. An enterprise also shouldn’t really have much in the way of pets (at the very least there should be a BC/DR environment!). Yes, if you have 1000 apps you might have 1000 playbooks, but each app team should be maintaining their own.

In a small business or a home environment, though, the case for full automation isn’t so clear. When each server is unique then maybe documentation of how a server is built, along with good backups to enable a restore in case of failure, might be a simpler and quicker solution.

But even in my pet-heavy environment there is still a need for a level of automation, just not full automation!

Bot scrapers DoS me?

Mon, 16 Jun 2025 10:53:54 -0400

I have a routine that runs every 15 minutes on my home machine and polls other of my servers and collates the results.

Once or twice a day one of my machines, at linode, was refusing to talk. It wasn’t causing a problem since the data is replicated and the system catches up, but it was annoying.

Digging around, the machine looked like it was working normally. But I found, in one log, that the machine had a load average of over 30 when the problem happens.

Now in the past when this occurred it was because of hardware issues at linode. But I wasn’t seeing anything in the logs. Normally sar data would show excessive I/O loads or long service times, but everything looked normal. Indeed it was just showing a higher (75%) CPU usage in user space, but still had over 20% idle time.

CGI

As luck would have it I was logged in while the problem occurred. A ps showed many dozens of CGI scripts running. Which is odd.

Checking my Apache logs and I found entries like

34.10.139.117 - - [03/Jun/2025:00:25:52 -0400] "GET /sf.pl?ACTION=SHOW&DETAIL=9780575050044 HTTP/1.1" 200 936 "https://spuddy.org/sf.pl" "Scrapy/2.11.2 (+https://scrapy.org)"
34.10.139.117 - - [03/Jun/2025:00:25:53 -0400] "GET /sf.pl?ACTION=SHOW&DETAIL=9780575052482 HTTP/1.1" 200 945 "https://spuddy.org/sf.pl" "Scrapy/2.11.2 (+https://scrapy.org)"
34.10.139.117 - - [03/Jun/2025:00:25:53 -0400] "GET /sf.pl?ACTION=SHOW&DETAIL=9780575055261 HTTP/1.1" 200 954 "https://spuddy.org/sf.pl" "Scrapy/2.11.2 (+https://scrapy.org)"
34.10.139.117 - - [03/Jun/2025:00:25:53 -0400] "GET /sf.pl?ACTION=SHOW&DETAIL=9780380756674 HTTP/1.1" 200 924 "https://spuddy.org/sf.pl" "Scrapy/2.11.2 (+https://scrapy.org)"

Well, that explains a lot. But I thought I had a robots.txt entry to stop this (after Google search indexer started hitting it).

Yup…

User-agent: *
Disallow: /cds.pl
Disallow: /sf.pl
Disallow: /video.pl

Looking at the web site for Scrapy I saw, quite predominantly featured, that it has a feature ROBOTSTXT_OBEY where this scraper can ignore the robots.txt entries.

That’s a pretty arsehole thing to build into your bot, to be honest.

Block by IP?

I originally thought of blocking by IP, but the scraper appears to be hosted at Google cloud and changes IP addresses.

In the past 2 weeks I’ve seen these sorts of volumes

% sudo grep  -h Scrapy *access_log* | cut -d' ' -f1 | sort | uniq -c | sort -nr
  74220 104.154.64.63
  60663 104.197.101.217
  54657 34.45.158.181
  50737 34.41.99.8
  50610 34.9.117.151
  49530 34.10.205.39
  38538 34.42.59.237
  38490 35.224.154.106
  30822 34.132.153.18
  23115 35.238.9.157
  23081 35.188.193.33
  15410 34.44.52.133
  15410 34.29.68.151
  15410 34.10.139.117
   7705 34.29.122.46
...

I’d be playing whack-a-mole if I went down this path!

Apache rate limiting with mod_evasive

I mentioned this else-net and someone responded they’d seem similar behaviour against their web server and had configured nginx to start rate limiting.

That sounded like a good idea. To the best of my knowledge, Apache doesn’t have this built in (I might be wrong! Let me know…) but it looked like a third party module, mod_evasive, could do the job.

I noticed that EPEL had this for RedHat 7, but not for 8 or 9. So I downloaded the SRPMS and used that to build an rpm for my servers using mock.

Config

After compiling and installing mod_evasive, I created this configuration (based on the defaults, all deployed by ansible).

LoadModule evasive20_module modules/mod_evasive24.so

<IfModule mod_evasive24.c>
    DOSHashTableSize    3097
    DOSPageCount        10
    DOSSiteCount        20
    DOSPageInterval     1
    DOSSiteInterval     5
    DOSBlockingPeriod   10
    DOSEmailNotify      sweh
</IfModule>

The results.

Initially I was worried about false positive blocks; would I start blocking legitimate spidering of my site (eg by google search). So each time I got a “blocked” alert I would log into the server and check it.

A few iterations of this, and some code changes (changing the response from 403 to 429; the email it sent out wasn’t quite right, and I wanted to enhance the data in the email so I could see if this was a good block) and I had something that seemed to mostly work.

Now I get an alert that looks something like

  Subject: HTTP BLACKLIST 104.154.64.63

  mod_evasive HTTP Blacklisted 104.154.64.63
      URI: /sf.pl?ACTION=SHOW&DETAIL=9780441662517
    Agent: Scrapy/2.11.2 (+https://scrapy.org)

When I looked through my logs yesterday morning I saw, in 2 hours

% awk '$9==429' sweh-ssl.access_log | wc -l
18460

Of those 17,940 were calls to a CGI.

Looking more closely at the entries, the first block was at “15/Jun/2025:05:19:17” and the last at “15/Jun/2025:05:21:49”. So it blocked 18,000 calls in 2 minutes. That’s pretty good!

Now it’s not perfect; I run Apache in prefork mode (‘cos I’m so old school!) so each forked instance keeps its own state, which means that if one web server process blocks a site another may let it through, or a process terminates and a new one replaces it then it will have no state and would allow traffic. eg at “15/Jun/2025:05:19:54” it blocked 262 attempts but let through 3.

But the number it blocks is really helping.

Who would have thought a cheap linode could handle 260 requests per second! Of course it’s all in memory when it hits 429 land, but still…

Forks of mod_evasive

I’ve since checked on github and I see there’s lots of forks of it. I spotted one of those forks appears to use shared memory to keep state so the prefork issue I’m seeing won’t happen. But that fork also removes some functionality I like.

For the moment, I think I’ll stick with this.

Side effects

Funnily enough this has also been catching script kiddies running simple bots against my site:

129.146.124.161 - - [15/Jun/2025:00:12:12 -0400] "GET /credentials.xml HTTP/1.1" 429 227 "http://spuddy.org/credentials.xml" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36"

13.74.158.147 - - [15/Jun/2025:14:33:13 -0400] "GET /wp-includes/Text/Diff/Engine/ HTTP/1.1" 429 227 "-" "-"

47.251.102.239 - - [15/Jun/2025:17:58:46 -0400] "GET /index.php?lang=../../../../../../../../tmp/index1 HTTP/1.1" 429 227 "-" "Custom-AsyncHttpClient"

It’s also spotted broken bots that keep requesting the same page (I’ve seen this one before)

117.159.55.9 - - [15/Jun/2025:13:48:21 -0400] "GET /images/emblem.gif HTTP/1.1" 429 227 "-" "Go-http-client/1.1"
117.159.55.9 - - [15/Jun/2025:13:48:22 -0400] "GET /images/emblem.gif HTTP/1.1" 429 227 "-" "Go-http-client/1.1"
117.159.55.9 - - [15/Jun/2025:13:48:22 -0400] "GET /images/emblem.gif HTTP/1.1" 429 227 "-" "Go-http-client/1.1"
117.159.55.9 - - [15/Jun/2025:13:48:22 -0400] "GET /images/emblem.gif HTTP/1.1" 429 227 "-" "Go-http-client/1.1"

And, of course, OpenAI

20.171.207.4 - - [14/Jun/2025:22:40:37 -0400] "GET /post/2018-01-04-meltdown_spectre/ HTTP/1.1" 429 227 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)"

Ranting

Web scrapers should obey the wishes of the content creator. robots.txt should be followed. At the very least they should act to not break the remote server. I know I can’t stop you stealing my stuff (and I’m resigned to that; theft of intellectual property is a different blog post entirely), but at least don’t break my machine.

Also you don’t need to scrape the same page a gazillions times. For example,

% cat *access_log* | grep -c 9780575050044
133

You’ve requested details about one book 133 times in 2 weeks? What the hell?

This is just abusive behaviour.

This web site is my hobby; it costs $20/month to run. It’s arseholes like this scraper that make this less fun.

Heh, at least I get the satisfaction of knowing that it’s costing you money to do this!

OK, rant over.

Summary

The modern internet is basically out of control. Mass theft of data for AI purposes is prevalent. I suspect that the majority of hits on my server are from bots, and not humans.

In the past 28 days, there were 771,000 hits on this web server (across the various URLs it serves), 548,000 were from Scrapy. Of the rest a quick check based on the user-agent, 55% were trivially bots (scrapers, script kiddies, RSS readers, whatever). Some of those bots may be useful :-)

Many of those bots are broken in various ways and can cause harm to victims.

In amongst this noise are various so-called “white hat” organisations that are also scanning machines without permission. At least they normally obey robots.txt and just cause noise in the logs.

It seems clear that most of the traffic on the internet is just machines talking to machines and not human eyeballs at all.

Any website, even hobby ones like this, are going to need more robust defenses against this brokenness. Tools like mod_evasive are reactive, but if they react quickly enough then they can prevent service degradation. You’d need different tools if you want to prevent data theft in the first place!

And an enterprise that needs to look at the logs to determine a real attack… good luck! All this brokenness and “white hat” traffic is just making your life harder.

I asked ChatGPT to write me a program

Tue, 10 Jun 2025 08:10:49 -0400

This post contains a lot of code, presented as close as possible to the code ChatGPT gave me. I’m including it here so people can see how good or bad they think it is. Where necessary I modified the code to make it work, but it’s as close as possible.

All this code makes the post look longer than it is. If you’re not interested in the code then you can just skip over it and just read my words :-)

The problem

I wanted to write a program to talk to the Fitbit API so I could get data out of it (eg record of heart rate while I was exercising). It’s not too hard to do; you just need an authorization token and a curl command

A quick hack’n’slash of code and I came up with something like:

#!/bin/ksh -p

d=${D:-$(date +"%Y-%m-%d")}

case $1 in
  cal*) T=calories; D=1min; I=activities-calories-intraday ;;
     *) T=heart ; D=1sec; I=activities-heart-intraday ;;
esac

URL=https://api.fitbit.com/1/user/-/activities/$T/date/$d/$d/$D/time/00:00/23:59.json

A="Bearer ....."

curl -H "Authorization: $A" $URL | jq -r '."'$I'".dataset[]|(.value|tostring)+","+.time'  | less

The problem is in getting the bearer token. This is oauth2 authentication. It could be done in shell script, but it’d be messy.

The proposed solution

Instead of doing this in shell script, I decided to do it in GoLang. I’d previously done oauth2 in perl (ugh, over a decade ago!) to talk to Google Calendar but I’m trying to use Go for stuff these days.

So I did some searching and Go has oauth2 libraries (naturally). It didn’t seem too hard. There were even examples of code talking to the Fitbit API. But I couldn’t find any code to explain how to use the refresh token. This is essential for automated use (e.g. from a cron job).

Let’s ask ChatGPT

So I thought I’d ask AI to help. Just the generic chatgpt.com, without a login; the bare bones generic free offering. If it worked then I’d be very impressed. If it didn’t then at least my biases would be confirmed :-)

Spoiler: it’s somewhere in between

I wasn’t going to “vibe code”, but use what I already knew to modify/fix what the AI gave me.

Getting the oauth2 token

So I asked it “I need a program in GoLang to perform oauth2 authentication from a CLI”.

And it provided a program that did this. I noticed it was designed for talking to Google endpoints, so I followed up with “what about for fitbit”. The resulting code was almost correct! Naturally it had places for hard-coded credentials, and the redirect URL wasn’t quite right; but these are things easily fixed.

After a little bit of cleanup (and me adding a quick kludge so I could make this a function) the code looked a bit like this:


package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"encoding/json"

	"golang.org/x/oauth2"
)

var (
	// Replace with your Fitbit app's credentials
	clientID     = "XXXXXX"
	clientSecret = "XXXXXX"
	redirectURL  = "http://localhost:8080"
)

// OAuth2 configuration for Fitbit
var oauth2Config = oauth2.Config{
	ClientID:     clientID,
	ClientSecret: clientSecret,
	RedirectURL:  redirectURL,
	Scopes:       []string{"activity", "profile"}, // Adjust scopes as needed
	Endpoint: oauth2.Endpoint{
		AuthURL:  "https://www.fitbit.com/oauth2/authorize",
		TokenURL: "https://api.fitbit.com/oauth2/token",
	},
}

var oauth2StateString = "random_state_string" // A random string for security purposes

// Function to handle the OAuth2 flow
func getOAuth2Token() (*oauth2.Token, error) {
	channel := make(chan string)

	// Step 1: Generate the URL for the user to authenticate
	authURL := oauth2Config.AuthCodeURL(oauth2StateString, oauth2.AccessTypeOffline)
	fmt.Println("Go to the following URL and authorize the application:", authURL)

	// Step 2: Set up the web server to handle the redirect and capture the authorization code
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		code := r.URL.Query().Get("code")
		if code == "" {
			channel <- "err_code"
			http.Error(w, "Code is missing", http.StatusBadRequest)
			return
		}

		// Step 3: Exchange the code for an access token
		token, err := oauth2Config.Exchange(context.Background(), code)
		if err != nil {
			channel <- "err_exchange"
			http.Error(w, fmt.Sprintf("Failed to exchange token: %v", err), http.StatusInternalServerError)
			return
		}

		// Step 4: Output the access token
		fmt.Fprintf(w, "Access token: %s\n", token.AccessToken)
		fmt.Printf("Access token: %s\n", token.AccessToken)
		channel <- "done"
	})

	// Start an HTTP server to handle the callback
	go http.ListenAndServe(":8080", nil)

	// Wait for the user to visit the URL and authenticate
	// time.Sleep(2 * time.Second)
	msg := <- channel
	fmt.Println(msg)

	return nil, nil
}

func main() {
	// Start OAuth2 token acquisition process
	_, err := getOAuth2Token()
	if err != nil {
		log.Fatalf("Error during OAuth2 process: %v", err)
	}
}

Now this actually worked! There’s a couple of places where it smells, though.

Hard coding the FitBit endpoints instead of using golang.org/x/oauth2/fitbit
No handling of the state string

But it worked. It got me a bearer token that I could use. That could easily be used to persist the returned data so it could be used later

How to use this?

So the next question was “how do I use this to get data?” and it returned some simple code. Again it worked. I’m not sure I had to mangle anything here.

package main

import (
	"fmt"
	"log"
	"net/http"
	"io/ioutil"
)

func getUserProfile(accessToken string) {
	// Fitbit API endpoint for user profile
	url := "https://api.fitbit.com/1/user/-/profile.json"
	req, err := http.NewRequest("GET", url, nil)
	if err != nil {
		log.Fatalf("Error creating request: %v", err)
	}

	// Set the Authorization header with the access token
	req.Header.Set("Authorization", "Bearer "+accessToken)

	// Perform the request
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		log.Fatalf("Error making request: %v", err)
	}
	defer resp.Body.Close()

	// Read and print the response body
	body, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		log.Fatalf("Error reading response: %v", err)
	}

	// Print the user profile
	fmt.Println(string(body))
}

func main() {
	// Replace this with the actual token you obtained during OAuth
	accessToken := "XXXXX"
	getUserProfile(accessToken)
}

There’s not really a lot to say there; it’s a simple enough http client.

Refresh token

But nowhere were we handling refresh tokens. So I then asked “How do we handle refresh tokens”

More code. This time it didn’t work properly because it was missing the basicAuth token (I had to add that). It’s also unnecessarily sending the client secret.

package main

import (
	"fmt"
	"log"
	"net/http"
	"net/url"
	"strings"
	"time"
	"encoding/json"

	"golang.org/x/oauth2"
)

var (
	clientID     = "XXXXX"
        clientSecret = "XXXXX"
	// Replace with your refresh token obtained during initial OAuth
	refreshToken = "XXXXX"
)

// Fitbit OAuth2 token endpoint
var tokenURL = "https://api.fitbit.com/oauth2/token"

// Function to refresh the access token
func refreshAccessToken(refreshToken string) (*oauth2.Token, error) {
	// Prepare the data for the token request
	data := url.Values{}
	data.Set("grant_type", "refresh_token")
	data.Set("refresh_token", refreshToken)
	data.Set("client_id", clientID)
	data.Set("client_secret", clientSecret)

	// Send the POST request to the token URL
	req, err := http.NewRequest("POST", tokenURL, strings.NewReader(data.Encode()))
	if err != nil {
		return nil, fmt.Errorf("Error creating request: %v", err)
	}
	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
	req.SetBasicAuth(clientID,clientSecret)

	// Perform the request
	client := &http.Client{}
	resp, err := client.Do(req)
	if err != nil {
		return nil, fmt.Errorf("Error making request: %v", err)
	}
	defer resp.Body.Close()

	// If the request failed, return an error
	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("Failed to refresh token, status code: %d", resp.StatusCode)
	}

	// Parse the response to get the new access token and refresh token
	var tokenResponse struct {
		AccessToken  string `json:"access_token"`
		RefreshToken string `json:"refresh_token"`
		TokenType    string `json:"token_type"`
		ExpiresIn    int    `json:"expires_in"`
	}

	// Decode the JSON response
	if err := json.NewDecoder(resp.Body).Decode(&tokenResponse); err != nil {
		return nil, fmt.Errorf("Error decoding response: %v", err)
	}

	// Return the new access token
	fmt.Printf("New Access Token: %s\n", tokenResponse.AccessToken)
	fmt.Printf("New Refresh Token: %s\n", tokenResponse.RefreshToken)

	// Optionally, update the stored refresh token, if Fitbit returns a new one
	// Update the refreshToken variable if necessary

	// Return the new token as an oauth2.Token
	return &oauth2.Token{
		AccessToken:  tokenResponse.AccessToken,
		RefreshToken: tokenResponse.RefreshToken,
		TokenType:    tokenResponse.TokenType,
		Expiry:       time.Now().Add(time.Duration(tokenResponse.ExpiresIn) * time.Second),
	}, nil
}

func main() {
	// Call the refresh function
	token, err := refreshAccessToken(refreshToken)
	if err != nil {
		log.Fatalf("Error refreshing access token: %v", err)
	}

	// Print the new access token
	fmt.Printf("Refreshed Access Token: %s\n", token.AccessToken)
}

It worked. But something is rotten in the state of Denmark.

Basically this is talking raw https to the Fitbit endpoints. It’s sending raw data, parsing JSON, doing everything itself. It’s not using the provided oauth2 libraries (except at the end, to create a token object).

This can’t be the right way of doing it.

The right(?) way

It turns out the reason I was having trouble finding a good answer, and likely the reason ChatGPT couldn’t do it, was because it’s not obvious. There’s a long thread on this going back 10 years.

It appears the original developers didn’t even think the refresh token could change ‘cos it’s optional in the spec and Google doesn’t seem to change it. So they embedded a refresh process inside the https communication flow and used the refresh token to generate a new AccessToken, but didn’t expose the new tokens to the caller!

Many of the solutions presented in that thread involve creating wrapper types with callback functions so the new tokens can be persisted transparently from the main code.

I decided to use a wrapper function instead, since I was going to build this type of abstraction anyway.

func call_fitbit_api(url string) []byte {
        // Get the oauth2 config and token
        conf := fitbit_config()
        token := fitbit_token()

        client := conf.Client(context.Background(), token)
        response, err := client.Get(url)
        check_or_die(err)

        check_refresh_token(client)

        body, err := io.ReadAll(response.Body)
        check_or_die(err)

        return body
}

And there’s a bit of magic where we delve into the client structure to find the current oauth2 token.

func check_refresh_token(client *http.Client) {
        // There's gotta be a better way to get the new token!
        nt, err := client.Transport.(*oauth2.Transport).Source.Token()
        check_or_die(err)

        if nt.AccessToken != configuration.AccessToken ||
           nt.RefreshToken != configuration.RefreshToken ||
           nt.Expiry != configuration.Expiry {
                configuration.AccessToken = nt.AccessToken
                configuration.RefreshToken = nt.RefreshToken
                configuration.Expiry = nt.Expiry
                save_config()
                fmt.Fprintf(os.Stderr,"Token refreshed")
           }
}

Digging so deep into a structure also smells a little, but at least it’s using the documented endpoints and I can’t find a better way. (If you know of one, let me know in the comments, please!)

Final code

The final code is on github.

Do note that this code doesn’t protect the token at rest; it just writes it out in the clear to the filesystem. This isn’t meant to be enterprise secure levels of code (you’d encrypt using the enterprise tools, persist in a store somewhere), but for my home use.

Now my shell script can become a lot simpler:

BASE=https://api.fitbit.com/1/user/-

get_activity()
{
  fitbit_get $BASE/activities/$1/date/$day/$day/$2/time/00:00/23:59.json | jq -r '."'$3'".dataset[]|(.value|tostring)+","+.time'  | less
}

case $1 in
   heart|"") get_activity heart 1sec activities-heart-intraday ;;
        cal) get_activity calories 1min activities-calories-intraday ;;
          *) echo Get what...
esac

Summary

So this both impressed and scared me.

It provided commented code that almost worked.

It’s the sort of code I might have hacked up 20 years ago (who am I kidding; I did hack up this sort of code 20 years ago!). And it would work. Well, mostly; there’s a potential race condition in the original refresh method depending on how it’s called; eg token.Valid() could return true but a second later the token expires so the GET request would fail.

But it’s bad code. It’s the sort of code that would worry me if it made it into an enterprise code base.

I’m not sure many AppSec tools would pick up on this either, since it’s a code quality issue and not a code security issue. I know there are code quality tools out there, but there are arguments that these don’t belong in AppSec but in AppDev. I also don’t know if they would pick up on this; the refresh code was OK as code when looking at it as a http client and JSON parser; it was just the wrong approach to the problem.

All this tells me that ChatGPT won’t replace good developers. We need them to apply the sniff test to determine if generated code is of a sufficient quality and where the approach taken by the generator is the right one.

That takes skill and experience.

I kinda wonder whether a programming copilot would have done better, but I’m not sure. This feels like a training data issue, and I couldn’t find good answers through regular searching.

VNC into an existing X desktop

Sun, 08 Jun 2025 14:57:13 -0400

When I was using a Mac as my media center player I liked that you could “remote desktop” into it; basically VNC and got the existing desktop. This meant I could use my local keyboard and monitor to control the machine that was 20ft away across the room from me, in the odd case where the command line wasn’t sufficient.

As MacOS went on these occurrences got more frequent… as did the BROKENNESS of the VNC server; I would frequently just get a black screen. sigh

But I’ve migrated away from MacOS to a Debian Linux desktop. I’m now using mpd as my music player (and have written go-mpd-music to give me a command line similar to what I was used to, and go-mpd-flirc to let me use an IR remote.

The problem was the remote desktop.

This turns out to be a hard thing for me to google, so I’m writing this up in the hopes it might help someone else.

vncserver

On Unix if you run vncserver then it will create a new desktop that you can VNC into. This is normally a good thing ‘cos it lets multiple people all connect to the same machine, have their own desktops that don’t conflict. You don’t even need a desktop X session running.

But that’s not what I want.

x2vnc

Totally not what we want; this lets you move your mouse and keyboard off the edge of the screen and it will translate the motion/presses into VNC events sent to a remote VNC server. In this way you can share your your keyboard/mouse between a Linux machine and another (eg Windows, running a VNC server).

x0vncserver

This is part of the TigerVNC package which may do what I want, but the description “an inefficient VNC server which continuously polls any X” didn’t make me want to look further.

tigervnc Xorg extension

Ah, now this is what I want. It’s meant to be more efficient than the normal “scraper” method and it’s an Xorg module so is loaded automatically when the X server starts.

Except it’s very poorly documented and it took me a while to find out how to deal with it.

In the end, on Debian, it turned out to be quite simple:

1. Install

The package is part of the Debian 12 repo, so we can simply install it.

sudo apt install tigervnc-xorg-extension

2. Configure.

Create /etc/X11/xorg.conf.d/10-vnc.conf with the following contents:

Section "Module"
  Load "vnc"
EndSection

Section "Screen"
  Identifier "Screen0"
  Option "UserPasswdVerifier" "VncAuth"
  Option "PasswordFile" "/etc/tigervnc/passwd"
EndSection

3. Create a password.

sudo vncpasswd /etc/tigervnc/passwd
Password:
Verify:
Would you like to enter a view-only password (y/n)? n
A view-only password is not used

4. Restart the X server

Probably easiest to reboot :-)

5. Check

After the reboot you can verify that the server is listening:

sudo netstat -anp | grep 5900
tcp        0      0 0.0.0.0:5900            0.0.0.0:*               LISTEN      1016/Xorg
tcp6       0      0 :::5900                 :::*                    LISTEN      1016/Xorg

Yup, port 5900 is the default VNC port!

And that’s it. You can now vncviewer yourmachine: and the native desktop will display.

Summary

This might have been easier if I knew the magic words to search for, but search results were full of entries around how to start a new X session under VNC, how to manage the window manager inside that session and the like.

It was only luck that found me a reference to this Xorg plugin!

What do I recommend? Generalist or specialist?

Wed, 07 May 2025 20:19:58 -0400

Back in 2018 I was asked about whether someone should become a Unix specialist.

In a similar vein, I saw a question on LinkedIn that asked whether someone should become a generalist or a front-end specialist or a back-end specialist.

Of course I had opinions :-) This is an extended take on my quick reply to the LinkedIn question.

Let’s scope this a bit better

I think, first, we need to think slightly wider than “front/backend”. A true generalist is more of a “full stack” person. By this I don’t mean they need to have knowledge of VLSI design, but knowing about CPU cache coherency might be useful. They don’t need to know about datacenter design, but being aware that some methods are less power efficient than others can help.

Definitely they should have knowledge from the platform layer up. So if you’re deploying to a Unix machine, having some Unix sysadmin skills would be useful; if you’re deploying to a k8s cluster then having some skills there helps. They don’t need to know how to configure a Cisco switch, but knowing TCP is a definite benefit. Being able to create database schemas, knowing when indexes are needed, monitor performance…

All these things are aspects a “full stack” developer should have.

Generalists

In my experience, I’ve found these people to be very valuable as part of a team. They may not be the best front end person nor the best back end person, but they know a bit of both. A good “full stack” person can talk to both those specialists, and the sysadmin, and the DBA and the network team and the firewall team and more. And, importantly, get those people talking to each other.

Because they have skills that overlap all these areas they can become team leads. They also may have a better overview of the project because they see how components fit together and how to do end-to-end analysis of data flow. Later may even become architects.

I’ve been that generalist. My first job kinda required it (managing Unix systems, networking, vendor relations, telex systems, telecoms…). Although I was primarily a Unix person I was also an old school code hacker; I wrote code in C, I wrote a complete shipboard general ledger system in TurboPascal, later I wrote the first commercial Java app sold in the UK (according to Sun Microsystems at the time, anyway!). Then later still an Oracle DBA, Window NT admin, Lotus Notes admin, web developer…

Having experience of all these things made me a more valuable person; for example, as a security architect I could see how things fitted and worked together; I could see the gaps and provide advice. I doubt I would even have reached that position if I had remained a Unix sysadmin.

We need specialists

But we need specialists. We’ve seen what happens when a developer tries to be a DBA (indeed one of the curses of the web is that it encourages this mess). The results are not as good, or sometimes miss some important aspect that a specialist would pick up.

So an employer needs both generalists and specialists on their team if they want the best outcome. They could get by with just generalists (less people to employee so cheaper), but the result may not be as good. Similarly, without the generalist the higher level architecture may suffer.

As time went on the technologies change; the DBA skills I learned for Oracle 8 still mostly apply to the latest versions, but there’s a lot more stuff that I’m not aware of. So, for example, when evaluating the security of ExaCS backups I relied on our senior DBAs to help fill in my knowledge gaps.

Similarly when looking at replacement Unix authentication tools I relied on senior platform engineers to do the hard design work for the infrastructure deployments. The generalist (me) saw how things should fit together (e.g. needing to enforce configurations, needing to integrate with the enterprise password vault, needing identity governance) and got all those teams working together; the specialists in each area worked out how to do the implementations.

A homelab could get me up to speed on some the technologies (especially if they’re free!) but it only got me so far.

Summary

There’s a methodology used by some to try and classify people; an I-shaped person is a specialist; they have deep knowledge in a specific discipline. A T-shaped person has a broader knowledge base but with a focus on one specific discipline. And then there’s M type people, who are multi-discipline.

I like learning new things and also getting people to talk to each other (“I don’t know that answer but you should talk to Fred Bloggs; he runs that stuff and will know the answer”). I’m an M-shaped person.

But I feel it’s essential to have I and T-shaped people in an organisation.

If I was starting over I think I’d follow the generalist path again.

But not everyone is suited for it, especially if you want to be “the best” at something. A generalist can be good at lots of things but there will always be people better than you in each of those areas.

Decreasing social media footprint

Tue, 25 Mar 2025 09:47:25 -0400

Leaving twitter

And my social media footprint shrinks; I’ve closed my twitter account, mostly ‘cos there’s very little left there to read. Yesterday there were only 5 or 6 new posts on the “Following” feed (I never used the algorithmic feed); 90% of the likes my posts got were bots. It really was the text equivalent of a post-apocalyptic wasteland with howling winds.

I had always been careful about what accounts I followed, so the extreme views that are reportedly on that platform rarely impinged on my consciousness; on the occasions I did click on a tweet it became clear that many of the replies were bots and misinformation or just plain hatred (“never read the comments!”), so I rarely did that.

I guess I’ve been on some form of social media (before it was ever called that) since 1987, when I gained access to Usenet, mailing lists, and (heh) online JANet BBS systems (Bradford UNaXcess, anyone?).

I remember the rush to LiveJournal when that started and then the mass exodus when it got sold to Russia. LJ was great in that it was a central place where pretty much everyone posted.

With the exodus came fragmentation; some people went to Dreamwidth, some to Facebook, some to their own blogs.

And that’s where I started to get tired of trying to keep up with people; some accounts I used to read on LJ fell off my radar (especially those who went to Facebook; I was never on that platform, never felt the need) and eventually became forgotten in the mists of time.

Twitter had its own “everyone there” vibe, but now it’s tumbleweed city with everyone scattered to diverse platforms.

And I don’t have the enthusiasm to start again on a new platform and then hunt down all the people I used to follow and…

It’s just not worth the effort!

Where am I now?

I’m trying to think what social media accounts I have left. Hmmm… Dreamwidth (which is pretty much just there to act as an RSS reader), LinkedIn, and… err. Does Reddit or Unix StackExchange count?

And there’s this blog, of course, but that’s not really social media.

How to contact me

For the few people who used to use twitter DMs to contact me, you probably have my mobile number and can text me (I don’t have encrypted messenger apps)… or you can find my email address on the About Me page!

So after almost 38 years of online activity my presence is shrinking and is now mostly this poorly maintained blog!

Five Years of COVID

Sun, 16 Mar 2025 20:29:53 -0400

So 5 years ago today was the day I told my boss I wasn’t going to come into the office for a while, and would work from home. Because I didn’t feel comfortable.

The company had made a plan; they were going to split the office into two groups who would come in alternate weeks. The idea was to reduce occupancy. However I’d been seeing more and more in the news how bad COVID could be and I didn’t want to risk being on the train for an hour each way as well as being in the office. After all I’d caught colds before and I knew Americans would come into the office sick (that’s a rant for another day).

All of my direct team was in different locations to me (Texas, California, North Caroline, Colorado, Georgia); it was really only a few secondary teams that I met in person (in particular the AppSec team; their manager tried to get onto the same weekly schedule as me ‘cos it gave us the opportunity for informal chats and collaboration). My access was via VDI and I had sufficient equipment at home, so my boss just said “OK”.

Now I didn’t expect this to be a long term thing; a few weeks, or a month or two. Then we’d all be back in the office. So I didn’t take anything home with me; personal stuff was still in the office.

Shortly afterwards I was told that I’d just been in close contact with a person who had COVID so I was meant to quarantine to avoid passing it on. So I felt my decision to avoid the office was the right one.

A few weeks later the company made most people work from home (there were some jobs that were considered essential and had to be done from the office, such as the print facilities and warehouse and datacenters, but most of us were meant to work from home. More justification for my decision.

I wondered if I should get a dog to keep me company, but still thought that this was just a temporary situation; it wouldn’t be fair on the dog to get used to me being at home and then be left alone all day when I finally returned to the office.

And then the company decided to close that location down. They were going to consolidate their New York/New Jersey locations into a new state of the art campus. On a town hall meeting about this they said they didn’t have any location in mind at present, but would make sure it was reachable by public transit because they knew people in this area generally commuted that way (especially those who worked in Manhattan and Jersey City).

Get a dog? Nah… I still thought I’d be back in office; maybe not immediately but within a year.

I rearranged my house so that the PC I was working on was no longer in my basement but moved up to the spare bedroom I was using as a library (bookshelves on 3 walls; table with PC on the fourth).

Then the new location was announced; and it was most definitely not public transit friendly. From my house it would need two trains and a bus and a lot of walking; maybe 2.5 hours each way. Hmm. No way was I going to commute there each day!

It took a while (longer than planned) for the new office to be ready, and eventually the “return of office” calls went out. I explained the problem to my boss (well, I’d already done this previously, so he was already on the case), who was accommodating and his boss (the CISO) was OK with it, so I became a permanent work from home.

I switched from VDI (which had been really glitching; it’s hard to be on a video meeting when your audio doesn’t work!) to a work laptop. And, honestly, it was so so much better. I knew our VDI was under-powered and over provisioned, but this was like night and day.

As you can see, this was a little cramped, but it worked.

Oh, and COVID! Yes! I got vaccinated! Because I’m considered obese, and thus at higher risk, I was able to get the vaccine. Spent a fair amount of time reloading reloading reloading websites to find somewhere with an open slot (“7 slots open! Sorry, all gone”).

I bought a bigger desk and two new monitors and a monitor arm, a better light and I got a lot more space. This was quite a comfortable work space.

And that’s kinda how I worked for the next 3 years. I got sent a second laptop (with Windows 11) for testing the build; I got sent a Macbook for testing (I’m not a security tester; I just got a reputation for breaking things so I would get sent this stuff) which made things complicated, but this worked.

I’m not much of a people person (very introverted) but I did miss the occasional chat with people in person, or going down the pub. But there was no way I could get to the new office.

I was fortunate; I had the space and I lived alone so I could set aside a room for work, and close the door at the end of work day; this meant I could keep work/life separation, which is vital for mental health.

And, eventually, after 3 more years of this (so 4 years of WFH) I retired.

And now it’s 5 years in. We’re no longer in a pandemic; COVID is endemic. This means it’s likely to be with us forever. It’s hard to know how many people are still dying of it, but wikipedia seems to think that 41,000 people died of it last year. That’s a lot; probably more than those who died in car crashes.

So I still get vaccinated every year. I don’t wear masks as often as I used to because most places have good ventilation, but I do wear them on trains or subways, or at the airport.

I haven’t had COVID yet, but a lot of that is likely because I am mostly at home, and don’t go to crowded places very often. I’m almost (but not quite) a hermit!

I never did get that dog; maybe I should have!

Why I don't use encrypted messenger apps

Tue, 25 Feb 2025 08:40:44 -0500

Secure messaging

A common question I get asked is “what secure messaging app do you use?” and the answer of “none” gets some surprised looks; how can I be in cyber security if I don’t use secure messaging?

The answer is “convenience”, with a side of “risk analysis”.

Back when Signal (on Android) did both secure messaging and SMS in the same app then I used this. When they removed this (because people might send insecure messages by mistake) I stopped using it. It’s inconvenient to have to wonder “Does Fred have Signal? Or WhatsApp? Or is it just SMS?”. I’ll just use the lowest common denominator.

Could my messages be intercepted? Sure! Am I worried? Not really. If the Chinese (or the US!) want to know that I’m planning a pub trip then good for them.

Same for email; I’m not using encrypted/signed email (PGP, S/MIME, whatever). Yes, my mail servers can do TLS but that’s “opportunistic” and they’ll happily fall back to plain text.

But.

I’m not a high value target; no one is going to go to the effort of grabbing my stuff. If it gets caught up in a high volume data theft (e.g. theft from a telco) then shrug nothing of value will have been taken.

There’s a slight caveat here; Google’s “Messages” app can do end-to-end encryption! When using RCS and configured and if the other party is also using “Messages” and is properly configured then the conversation is encrypted. At the moment it’s limited to Android users using Messages, but since Apple is now embracing RCS maybe we’ll also see some cross-platform support in the future?

VPN on my phone

Nope, don’t use that either. If I’m in Starbucks my phone is using mobile data directly with no VPN to protect me. But I’m not concerned because pretty much every site my phone talks to is TLS protected. This gives encryption and a level of identify verification (“yes, this is sweharris.com; the certificate says so!“).

I don’t normally have WiFi turned on when I’m out of my house (automated based on location) but even if I did and connected to a rogue hotspot then I’m not too concerned; the traffic is still protected through TLS. Sure the bad guy might see my phone make a connection to my bank, or to my home network; they might be able to see my DNS queries. So from this meta data they might be able to work up some sort of profile on me to use in social engineering, but my data is secure.

Again a minor caveat; I do have a VPN I run myself, but that’s normally off; it’s to let me access my own IMAP server if I’m out of home and need access to my email urgently. Mostly I don’t need it. And I don’t default-route via the VPN, just my home subnet, so everything else would still go out the normal way.

VPN at home

Why would I? Again, most things are TLS protected. I run my own DNS servers (ISC Bind) so I don’t hit my ISPs servers for that. Sure they could sniff my DNS traffic or they could see the TLS headers and so learn that I spend far too much time looking at pr0n. But so what?

Yes, Verizon would release my details to the government on request, but then so would most VPN providers. I’m not gaining any security, just moving the risk around.

And getting worse network performance as a result.

There are use cases for VPNs (especially if hoisting the Jolly Roger) but security isn’t one of them. So, no, I don’t use a public VPN.

(As above, I do have a private VPN I run myself that I use to allow services out of my home to reach into my private network)

Anti Virus

I do now have a Windows machine; I use it for gaming. But the only protection on it is the built in Microsoft Defender product. I’m not hitting warez sites, downloading dodgy software; I don’t even have Office products installed so no Word Macro viruses here! I don’t read email on this machine (I do that on Linux using a terminal client, “mutt”). Basically the risk vectors for attacking and infecting this machine are very very small. So Defender is good enough for me. Honestly, it’s probably good enough for most individuals.

Alexa and smart home stuff

Yes, I’ve got spy bots in pretty much every room in my house, even in my main bathroom (I use it for morning news from NPR and BBC, and to play music while I’m showering).

No, I’m not concerned about Bezos listening in on me.

I have a lot of home automation and having voice control over them is nice. My MQTT server is not password protected, but you have to be on my LAN or IoT networks to see it (guestnet can’t).

A rant

And here is where I’m gonna rant about mobile email; because of the form factor it’s very hard to get some of the information needed to determine if an email is legitimate; it may not be easy to tell Fred Bloggs <myfriend@their.email> from Fred Bloggs <badguy@evil.badguy> on a mobile email app; they both show as “Fred Bloggs”. Grump! And a similar rant for embedded browsers in chat apps; they may not show URLs!

So we need to be very aware of attacks on the human aspect; don’t open random attachments, even from friends. Don’t enter your password into a site you’ve reached from a link in a chat message. And so on. Technology like secure messaging or VPNs won’t protect you from that sort of stuff, anyway!

Summary

Most of the things I don’t do are because of risk evaluation and what I consider valuable. Of course if I was negotiating a multi-billion dollar deal then I’d want to do it securely, but “wanna go down the pub tonight?”… not so much. If I was accessing corporate resources then a VPN or Zero Trust tunnel or similar would be necessary. But to access my bank account? TLS is good enough.

Everyone needs to look at their personal circumstances and decide what level of protection they need.

I honestly think that, for most individuals, the out-of-the-box technology protections are now pretty good. The bigger risks are “human factor” related, such as phishing or scams.

Stop thinking of privilege in technical terms

Fri, 24 Jan 2025 09:14:47 -0500

I recently saw a posting on LinkedIn that said something like “with zero trust we can consider all access as privileged access”.

While this could be considered true, I also made the same argument 15+ years ago before zero trust was a thing people cared about; my argument was “if I can login to a server then I can run commands, impact applications (eg chew up CPU), fork bomb, etc; surely that means all access is privileged”.

Now while that is true enough, it’s not necessarily a helpful stance. The term “privileged access” has a lot of connotations, especially in regulated industries where auditors want you to have strong controls in place.

After all, with zero trust in place your personal access your email is something that’s evaluated and permissions need to be in place. We don’t always consider this to be privileged.

Technology view

So businesses may next start thinking about how privileged access may work.

On a Unix machine having the ability to do things as root is normally the first place we end up. If the company is smart they’ll use something like sudo (or equivalent) to provide some finer grained control. Users log in with a low privileges and then use these tools to escalate their privileges; a user that can do this is a privileged user. And then the control expands to cover the oracle account, the service/functional account used to run your app, and so on.

On Windows we might deploy a privileged escalation tool to do similar to what Unix does, or we might have “admin” accounts in a privileged group and put those passwords in a controlled vault. On Oracle we might do similar with DBA privileges.

But in all of these solutions we’re looking at this from a technology perspective; if the platform considers activity privileged then we classify access to this as privileged.

Weakness of this

The problem with the technology view is that it brings a lot of people into scope for stricter controls (e.g. more frequent recertification, enhanced activity oversight) than is necessary. It also frequently leads to “laziness”; oh if I’ve got to jump through these hoops then I might as well just do sudo su.

A simple example is from Solaris; on this platform (at least Solaris 8 and before), ifconfig -a will not show the MAC addresses of the interfaces; you need to be root to see that information. So now if you grant someone sudo ifconfig -a (being careful to ensure that’s all they can do!) you’ve granted someone a technology privilege, but are they really a privileged user?

Another example might be the df command; if a mount point is hidden then the disk may not show up in the output, but if run as root it will.

% ls -ld /tmp/dir1
drwx------ 3 root root 4096 Aug 26 15:42 /tmp/dir1

% df | grep dir1

% sudo df | grep dir1
/dev/shm          249720       0    249720   0% /tmp/dir1/mountpoint

Is this really privileged access? Knowledge that the mountpoint exists may be exposed elsewhere (eg /proc/mounts); we really only need elevated privileges because of how df works (it can’t do a stat() on the mountpoint because the parent directory doesn’t allow the user access).

Business privilege

So let’s look at this from a business perspective, instead.

The goal of all these controls is to provide the CIA triad; confidentiality, integrity, availability.

We need to evaluate the risk of activities under this umbrella, and what is considered an unacceptable risk to one business may not be one to another.

We might decide that sudo ifconfig is privileged because you can make changes (e.g. change IP address) but sudo ifconfig -a is unprivileged because it’s not exposing sensitive data. This is a business decision.

So now our definition of privilege can be rewritten in business terms; perhaps something like “activity that exposes or modifies sensitive data, or impact the running of the business”.

OK, that’s pretty weak but it’s a starting point; a customer can access their own data (eg bank transactions) but that’s not considered privileged. However a bank teller can access multiple people’s transactions and so is privileged.

So this definition can (and should!) be refined. This is potentially hard.

But it’s important to note that the way we’re defining this isn’t in technology terms, it’s in business terms.

Now we can look at access rules and determine if they’re business privileged or not. We can take a risk based approach.

Of course this does require help from the platforms we’re trying to control; the finer grained the controls can be, the better we can limit access, and the fewer people need to be considered “privileged”.

Entitlements vs privilege

In order to advance the discussion we need to change the terminology a little as well. In the opening statement (“all access may be considered privileged”) we’re really conflating entitlement with privilege.

An entitlement is the ability to do something (read your email, get your payslip from Workday, login to an application…)

A collection of entitlements may be consolidated into a role. e.g. “Unix system admin” may be a role that has entitlements of “can login as themselves”, “can sudo df” and similar. People will have multiple roles, and some of what are sometimes considered “birthright”; e.g. as an “Employee” you automatically are granted “Can read email”, “can get their payslip”.

Note that these are business roles and not technology roles. Windows has roles exposed as security groups (eg “Backup Operators”) which are collections of entitlements (eg “Backup files and directories”, “Restore files and directories”). It’s important not to consider these as roles from a business perspective; this is just a group of entitlements.
Note that entitlements and roles may be constrained in scope, and the larger the company the more likely this will be; e.g. “Unix system admins in North America” can not login to EMEA systems. This is why it’s important to distinguish business roles from technology roles even if they have the same name.

A role may be considered privileged if it allows members of the role to perform a business privileged action; “Bank teller” is a privileged role because members can see transactions for multiple people.

Finally a person is privileged if they are a member of a privileged role or if they have a business privileged action directly assigned to them.

So now we could validly argue that the first statement isn’t really accurate; not all entitlements are privileged so not all access is privileged.

This all comes back to the definition of “privilege” as a business function not a technology function.

Conclusion

This is hard work.

Defining and maintaining these roles isn’t easy; evaluating every entitlement to determine if it’s business privileged or not (or if the business accepts the risk of considering this non-privileged), whether the technology is capable of creating sufficient fine grained access is hard and time consuming. Role management, in particular, is very hard to get right.

Even if you don’t define privilege along business lines, thinking in business terms (business roles, instead of technology roles) can bring benefits; you gain some ability to provide minimum necessary access to systems, you can reduce audit overhead, access requests become simpler, access is more normalized and can be automated easier.

These benefits also apply to zero trust environments (indeed the rules may be made simpler) and to zero standing access setups (which can control if the entitlements are currently granted; your role determines what entitlements are available).

Comparison of all my screwdrivers

Sat, 11 Jan 2025 15:10:53 -0500

This post may seem odd for this blog; after all, why would anyone be interested in my screwdrivers? After all, someone like Project Farm did a scientific(ish) comparison of various things and gives you a lot more data than I ever could.

But we’re all human, and sometimes a subjective opinion is valuable. And as people know, I have opinions :-) This may seem long but if you just want my opinions on the LTT drivers then skip to the bottom.

Anyway, one day I was re-organizing my cupboard when I realised that I had a fair number of screwdrivers in my toolbox, loose on shelves, in stacks in the cupboard.

And then LTT had a deal on their precision driver and bit set so I got one of them. Then later they had a deal on their ratcheting screwdriver so I got one of them as well. I didn’t need them, but I thought it’d be interesting to compare the hype against what I already had.

So now I have a lot of screwdrivers!

(most images can be clicked for larger versions)

Stanley

When I moved to America in 2001 I didn’t bring any tools with me. After all, the electronic stuff (eg drills) wouldn’t work, and the rest I could get easily once I was there.

Just around the corner from the office where I worked was an old school hardware shop. I would sometimes pop in there at lunchtime to get stuff I needed.

The first kit I got were these Stanley drivers

What I hadn’t realised when I bought them was that two of the heads were duplicated, just on longer shafts. I tried comparing the head sizes to other (labeled) heads, and I think this is what was in it.

1/4" (SL7) with 4" and 6" shafts
3/16" (SL5) with 3" and 6" shafts
PH2 with 4" shafts
PH1 with 4" shafts

They’ve last well; some cosmetic marks near the tips but, other than the limited head selection, they’re just fine. And I’m not afraid to hit the handle with a hammer :-)

Old school precision drivers

Next I needed some precision screwdrivers. Same store had these super duper cheap sets.

Ooh, fancy; tweezers and a magnifying glass!

PH1, PH0
1.4, 2.0, 2.4, 3.0mm flat head

The lid hinge died pretty quickly, but duct tape to the rescue.

The biggest annoyance with these is the handles somehow fail to give the necessary grip on really tight screws and it can cause abrasion on your fingers as you try and grip harder and harder to undo that stubborn thing.

Some oddities

Stubby

At the same time I saw a stubby screwdriver; I wasn’t sure I had a need for it, but for a few bucks it was worth getting. The shaft is removable and flippable; one side is a PH2 and the other is SL7

Bits in the handle

Also from this hardware store came my first “true” changeable heads driver. In this one the bits were stored in the handle. It’s next to the stubby in the main picture. The closure for this was a simple plastic clip that was hard to open/close… and eventually snapped off. So now the handle won’t stay closed. But it kinda still works, and was convenient! Pozidrive, rather than Phillips bits. Huh.

PZ3, PZ2, PZ1
SL6, SL5, SL4
socket head adapter
10mm, 9mm, 8mm, 7mm, 6mm hex socket

Helping Hands

I was buying a new house and needed a screwdriver quickly, so went into the local CVS and bought this ³⁄₁₆” (SL5); clear handle with blue stripes.

Yellow flippable

Pretty sure this came with some electronics kit (maybe a USB drive bay for 5.25” disks?). The ends aren’t very accurate in their sizing and it kinda looks like SL3.5 and somewhere between PH00 and PH000.

Curtis kit

Pretty sure this also came from the same hardware store. It was an interesting combination of things; a ratchet handle (with a grip that allowed more torque), a set of changeable tips, an extension shaft, a pocket-clip small driver, tweezers, spring claw, pocket flash light, electrical tape… such an odd selection!

The tool heads were also pretty varied; I didn’t have any Torx before. So…

SL6, SL4
PH2
PZ1 
T25, T20, T15, T10, T8
S3, S2, S1, S0
socket head adapter
12mm, 11mm, 10mm, 9mm, 8mm, 7mm, 6mm, 5mm hex socket

This kit lasted well, and finally I had ratchet! The most annoying part was how the heads were stored. Getting them out of those rubber holders can be a pain, and the holder covers the marking that says what size the head is.

Already I was starting to get duplicate drivers.

Ratcheting drivers face off!

Megapro Automotive R

I still wanted a “real” ratchet driver; the Curtis one was OK but the wide handle and short shaft often meant I couldn’t use it when near the edges of something. I also didn’t trust it for real heavy duty stuff.

So I got this.

This feels heavy in the hand; it feels like it means business. And it’s labeled “Automotive” so, in theory, it should hold up to hard work.

The ratchet mechanism is smooth (I spent the next few days just playing with it as a fidget toy!). The hat that holds the bits is a bit hard to open but it moves smoothly. The handle has rubber strips and knobbly bits to maintain a good grip.

The provided heads are clearly also designed for heavy duty work; they’re large!

PH3, PH2, PH1
S2
SL6, SL4
T30, T27, T25, T20, T15, T10

These heads clip into the bit holder very easily and the markings on them are easy enough to read.

It’s a standard ¹⁄₄” fit with a pretty strong magnet so I can use the tips from the other drivers.

LTT “Retro”

I wasn’t going to get this, but LTT were having a “mystery driver” sale. I ended up with the “retro” driver. And it’s clear how this was influenced by the Megapro, but is different.

For starters, the LTT driver is smaller; the shaft is a little bit smaller, the handle a little bit smaller, the bits a little bit smaller, the storage hat a little bit smaller. This makes it lighter.

There are other changes as well; the handle is a new shape. It’s not rubber grippy like the Megapro, but has more of an angular design to help maintain grip. It also has a smaller neck which my middle finger falls naturally into. The storage hat is also easier to open and has extra “grip” areas to make it even easier.

Unfortunately either it’s the smaller bits (they’re about 5mm shorter), or the plastic used, makes pushing the bits into the storage holder a little bit hard. I can only hope this plastic doesn’t get brittle as it ages because I feel it might be a weak point. I could be wrong, of course :-)

The bits also are harder to see the markings on. The black coating really makes it difficult; you have to angle them ‘just right’ in the light to see the size.

The other big change is on the ratchet itself; this is a little lighter. Along with the knurling it’s a lot easier to use this driver one handed when the screw doesn’t yet have enough friction to let the ratchet work. I can even turn the shaft with my thumb, alone; something the Megapro can’t do. For some reason they decided to reverse the direction that the ratchet selector works; in theory the LTT way is more natural since it matches the direction you will turn the handle, but it’s opposite to many others and so is a little confusing.

The bit set load is definitely more suited for computer work.

SL6, SL4, SL2
PH2, PH1, PH0
S2, S1 
H4, H2.5, H2
magnetic screw catcher

The extra size available on the megapro also allows it to have a separator in the head, which I feel also helps manage the bit; the LTT storage head skips that to reduce size and weight.

Precision drivers face off!

Tekton Precision kit

The old school precision drivers I had were annoying me; I spotted this on Amazon; TEKTON 28301 Tech Rescue Kit.

This is a hell of a kit packaged into a small soft-shell case. It comes with a LOT of stuff!

As well as the magnetic screwdriver shaft, very fine nosed tweezers, a metal spudger, a nylon spudger, a suction cup puller, and a magnetic screw holder, it has a large selection of tips

2.5mm, 1.5mm, 1mm slotted heads
PH1, PH0, PH00, PH000
PH1, PH0 3" shaft
T5, T4, T3 (star bits)
TR15, TR10, TR9, TR8, TR7, TR6 (tamper resistant star bits)
TR8, TR6 3" shaft
TRI1, TRI0 (tri-wing)
2mm, 1.5mm, 1.3mm hex
TA27, TA23, TA20, TA18 (triangle)
PL1.5, PL1.2, PL0.8 (pentalobe)
6.5mm, 5mm, 4mm, 3.5mm, 3mm nut drivers

This is a great kit, and very cleverly put together; the bits are easy to remove from the plastic holders, which are clearly marked, and they are held in the case with velcro strips. The driver handle is knurled to provide a good grip.

The biggest problem I have with this kit is that I was tempted to use it for things I shouldn’t have; it’s a precision kit, not a general purpose kit!

MegaPro 251Precision

Since I liked the larger MegaPro so much, I thought I’d try their precision driver. It wouldn’t replace the Tekton kit, but it might be useful as a more portable “just need a screwdriver” tool.

Unfortunately the storage hat is about all it has in common with the ratchet. The heads are all “flip” style (each head has two tips) and they’re not easy to get in/out of the shaft.

The selection of heads is good…

T15, T10
S3, S2.5, S2, S1.5
P2, P5
SP2.6
TA 2.3
TX5, TX4, TX3
TT15, TT10, TT8, TT6
PH2, PH1, PH0, PH00, PH000
SL3, SL2.5, SL2, SL1.5

Unfortunately because of the double-sided nature they can’t be organized in a sane manner so I’m always searching for the right one.

LTT Precision

This was combined deal; the driver and the bit set together. It comes in a storage box and the lid can be removed and is magnetic with compartments so screws can be held together; especially useful when some kit has different sized screws!

The big hype from LTT about this driver was the spinning hat; so much time was spent talking about how it can be a fidget spinner and it runs for a long time and… yeah, OK, it’s smooth, it’s clever. But this is a screwdriver. Unfortunately the storage head can only hold 3 bits, so this is very unlikely to ever be used “on its own”; all the bits will live in the storage box along with the driver.

The box is much bigger and heavier than the Tekton kit. If you’re gonna throw it in a toolbox then it’ll probably last longer, but my use case is less demanding; the smaller soft shell case wins.

There’s a lot of bits in this set and the base of each bit is smaller than traditional; this lets them use a longer shaft on each one, which might make it easier to get into hard-to-reach places.

PH2, PH1, PH0, PH00, PH000, PH0000
SIM card ejector
P6, P5, P2
SP8, SP6, SP4
PZ1, PZ0
S2, S1, S0
TR26, TR20, TR15, TR10, TR9, TR8, TR7, TR6
T5, T4, T3, T2 
Standoff remover
TA3, TA2
Y1, Y0, Y00, Y000
SL4, SL3, SL2.5, SL2, SL1.5, SL1
H5, H3.5, H3, H2.5, H2, H1.5, H1.3, H0.9, H0.7
Magnet
hex socket 5 4 3.5 3 2.5
g socket 4.5 3.5
adapter to allow these to be used in a 1/4" driver

As with the LTT ratcheting driver, it’s really hard to read the bit labeling; it’s embedded into the foam of the storage case so there’s very little contrast and hard to read. The bits, themselves, are easier but the marking is hidden when they’re in the case!

Summary

So that’s it; that’s all my screw drivers!

Going forward I’m likely to use the MegaPro ratchet driver for “tough” work, and the LTT ratchet for computer stuff. I guess, in theory, I could swap some of the LTT bits into the MegaPro storage hat, but keeping them separate is probably easier.

For precision work I’m conflicted. The shape of the LTT handle is a little bit better than the Tekton one. But I’m also likely to want a spudger when I’m taking things apart, so the LTT kit on its own just isn’t sufficient. I’ve tested and I could just swap over the handles; use the Tekton bitset with the LTT handle. That might be thing to do.

Unfortunately I’m not sure the LTT stuff is value for money.

The Tekton kit cost me $27 from Amazon; the LTT precision kit (including tax+shipping) was $64. Yes, the LTT kit has a lot more bits, but its missing things needed to make it a “complete” kit.

Similarly the MegaPro ratchet was $37, the LTT was $53 (which is cheaper than normal because it was on sale; $40 vs $70, the rest being tax+shipping). This one is harder to pick; they’re both good drivers, but the bit set on the LTT one is more suited for computer work. Maybe there’s a different MegaPro with a different bit set that is closer!

I do wish LTT would be better with contrast on their bit case and bits, though! For someone who can’t tell the difference these labels can be very helpful.

We don't need security products

Wed, 11 Dec 2024 19:54:02 -0500

There’s a theme going around that you should create secure products, not buy security products. And, as far as it goes, this is…

Well, actually it’s not good.

My initial response was “Why not both?” We need to secure the products we develop. There’s no doubt about that. And we need to mitigate mistakes. How do we do this? Spoiler… security products :-)

In response to this I got a message “If you have secure products, you do not need security products. And what you do not need, you definitely should not run!”

And at that point this rant was generated.

Rant

See, how do you know your product is secure? ‘Cos as sure as the sun will rise tomorrow (even if I might not see it ‘cos of the awful weather…) a developer will make a mistake; a core library will have a bug (hi, OpenSSL!), a service will be misconfigured, a subsystem will be left unsecured.

Is your code provably correct? Really?! Does it depend on any external libraries? Are those libraries secure? Will they remain secure? How many new CVEs are generated against code we consume as part of our development processes, and that may make our own code bad? We may write the most perfect code ever, but a vulnerability in a dependency can ruin it all.

And we all have dependencies in our code bases. For example, we sure don’t want developers creating their own encryption systems; that’s hard enough to get right for seasoned cryptographers (see, for example, some of the failings of the newer post-quantum algorithms; see, for example, how older algorithms are deprecated as attacks are discovered). But we even have more basic dependencies; how many maven libraries is your code bringing in? How much stuff has been cloned from github?

And the inescapable truth is that developers are human (and “AI” assistants are sub-human). Humans make mistakes.

So what do we need?

Obviously we need to help our developers create better code. We want to improve the skills of our developers and make their code better. So let’s alert them to mistakes they might be making. A plugin in their IDE? Some static code analysis? Hooks into the devops process to block bad code from being promoted? Even some training modules focused on discovered weaknesses? They’re all security products.

What about those dependencies? We need to track what modules a developer is using and ensure they have no known vulnerabilities or (better) that any known vulns aren’t in the execution path and so not impacted. So now we need tools that can do dependency analysis, compare what’s being used to what’s good, and preferably block bad modules from the build.

Oh, that module which was good last month… it has a new CVSS 10 vuln; do we know where that’s been deployed so we can target remediation? More tools.

What about identity and access management? What about new classes of attack (e.g. API issues)? What about…?

Security tools can even provide mitigation against lower level attacks, such as a DoS. Or provide protection for third-party tools. Do you put your ILOMs and DRACs and every other out-of-band interface on the internet? Or do you limit them - eg via subnet routing, firewalls, etc. Require a VPN with MFA to reach them… look, more security tools!

Reality

The reality is that we strive for perfect code. We all want to do the right thing. But we all can fail. The larger the product, the more developers, the increase in complexity… the greater the risk. And modern microservice architectures are complex.

Even the mythical 10x developer makes a mistake every now and then.

So we create standard architectures to try and limit the risk. We use security tools to provide “belts and braces”, to provide depth, to mitigate mistakes that will happen.

Sure, don’t run what you don’t need. But you do need the right security tools. You have limited resources, so pick wisely!

Stop changing technology

Sun, 20 Oct 2024 12:56:18 -0400

One thing I’ve noticed, over the years, is the habit of people blaming technology for the problems rather than taking a look at the processes behind the problem.

A personal example

A big example, for me, was when I was part of the Unix enterprise authentication team. The technology worked, and it worked well. It was resilient, reliable, fast. We literally turned off the infrastructure in one datacenter and all the clients correctly failed over to the next nearest one.

But it had a bad reputation. If you so much mentioned the word “Keon” to some people then their eyes will start to twitch and they get a haunted look.

Because the processes around configuring the environment were bad. To build a new application you needed to think ahead and plan your requirements; “these are my servers”, “these are my app team members” and “they need these Unix permissions” (I know a lot of people don’t like this, but I thought it reasonable; we can’t put servers on the network without access controls).

But it got worse. You had to fill out a spreadsheet using complicated jargon and half a dozen tabs, just to say this. Then you had to attach this to a “consulting request” in the corporate request system. With a 5 day SLA and a high likelihood of the request being rejected.

How do I know the process was bad? Because my request was rejected; not because it was wrong but because the implementer missed the attachment!

So when a new head of technology joined the company they heard all these complaints about Keon and how it was so bad, and he planned to replace it.

I just couldn’t convince him that replacing the technology stack would not fix the processes; it was the process that people hated. Focus on that.

Unfortunately the process side of the house wasn’t in his remit (that was run by a different technology tower) so instead of working with that tower (we could have created better tooling for the request system that would have made both the requester’s life and the implementer’s life better) he treated it like a technology problem and tasked me with evaluating replacement tools.

(I soon moved out of that team and into the new cloud team)

Iterated madness

“The definition of insanity is doing the same thing over and over and expecting different results” – Albert Einstein

Elsewhere I’ve seen teams get stuck in a loop; they would buy the best in breed technology for a solution and deploy it. It would fail to meet the requirements, so they would go back to the drawing board and get replacement technology. Rinse repeat.

The underlying cause is that they were looking at this as a technology delivery problem, and not as a service delivery problem. The team were technologists, so this kinda made sense; they were told “deploy X” and they did.

So the blame lies higher up the management tree; the VPs and SVPs who didn’t see the bigger picture, who didn’t understand the problem they were told to solve, and who believed the vendors.

The role that was missing was one of “service owner”. What do we need to do to make this a consumable service? They could have saved millions of dollars and reduced technology disruption just by having this bigger picture.

Service and Product owners

There are multiple ways this can be defined, but I look at it this way.

The Product Owner is the one that owns the technology stack. They’re the one who is responsible for keeping it running, for ensuring it’s patched and updated, for understanding the vendor technology road map for the product, and for handling any integrations with other systems (request system, change management, HR, etc).

The Service Owner is the one that is responsible to the users of the product to ensure the business needs are met. They create the overall service road map that is informed by the vendor road maps, but may also takes into account customer complaints, service issues, etc. It may be the vendor product road map includes sprouting wings and flying, but if you want to dig tunnels we may not implement that function… unless your service needs include rapid deployment to remote locations :-)

These two owners should work tightly with each other; they may even be the same person, but need not be.

Each role is critical to the success of the service.

Summary

“Technology is easy; process is hard” – me

We deploy tools to meet a business need, not for the sake of deploying them. These tools are not standalone; they have inputs and outputs, even if it’s just reporting to the CISO about what they’re doing, how they’re reducing our risk, and how well they’re achieving those goals!

So we need to look at this as a service and not a technology. And when the service fails we need to look at it from a service perspective. Is the technology to blame (it could be!), or are we using it wrong? Even if the technology is wrong, is it because we didn’t deploy it correctly and it can be fixed?

Don’t just change the technology because it’s easy (for some values of “easy”); look wider. Find where the problem really is.

The problem may be that you have the wrong people; it may be that you have the wrong processes; it may be that you don’t actually understand the problem!

Replacing a technology stack will only fix the problem if the underlying technology is wrong for this problem space, and if you’ve bought the best of breed then it’s not likely. You may have deployed it wrong (fix that) or aren’t using it properly (ditto) but if you just replace it and hope this time things will work out better… well, that’s Einstein’s definition of insanity.

Google killing adblockers

Thu, 12 Sep 2024 08:33:47 -0400

Google has been threatening this for a while, but now they’re finally getting around to it; they’re starting to remove Manifest v2 (MV2) from Chromium (and thus Chrome, and likely many browsers based on chromium, which is the majority of the browser space, these days!).

What does this mean?

Chrome extensions use a set of APIs to talk to the browser engine. The main version that’s been in use for a number of years is “Manifest v2”. This set of APIs are extremely powerful and flexible and allow an extension to modify almost every aspect of your browsing experience. Extensions such as uBlock Origin (uBO) and NoScript make use of this to inspect web pages and modify them; e.g. to remove adverts, block JavaScript and so on.

Google are removing this API and replacing it with a newer one (“Manifest v3”; MV3) which is more limited in capabilities. This will cause a number of extensions to break, and may make it infeasible for them to operate at all.

If you are on the beta, dev or canary channels then you’ll already be seeing impact; it hasn’t yet reached the main stable channel, but it’s only a couple of major releases away.

Why are they doing it?

Google’s primary claims are around security and performance. If a rogue extension was deployed to your browser then with v2 it can see pretty much everything; it could steal your bank login credentials (so always use MFA!) or insert adverts or… anything. This is obviously bad. By enforcing a more limited API the risks are mitigated to some extent.

Similarly extensions (even ones you want) can be slow because they can do so much. An extension with thousands of rules could slow down every web page you go to, and take up lots of memory. Chrome frequently gets the blame for being a memory and CPU hog, but how much of this is the core browser engine and how much is due to the dozen extensions people load?

So what’s the problem?

Manifest v3 has strict limits which prevents uBO from being ported over. Instead, the author has created a uBO Lite. This has some functionality changes and a lot of limitations.

Possibly more importantly, filter rules are no longer updated from the internet but are embedded into the extension. The rules are now only update when the extension is updated, and this may take some time as it goes through the Google evaluation processes. This could mean that new advertising solutions may get through until the extension is updated; with MV2 the extension could regularly update its lists. In the past I’ve seen sites such as twitter modify how ads are displayed, but then a few hours later uBO rules update to handle this.

And this is why people are worried about MV3 and why Google’s rationale isn’t as persuasive; the feeling is that Google (the owner of the world’s largest digital advertising platform) is deliberately trying to hobble ad blockers.

We’ve seen ads used to propagate malware in the past, and even NIST recommend “Use advertisement blocking extensions” as part of safe web browsing, so there’s also a concern that Google are reducing overall security; an ad blocker could mitigate a Chrome zero-day attack!

What can we do?

For the immediate future, keep an eye on your extensions. If they get disabled you may be able to temporarily re-enable them.

It’s definitely worth trying out MV3 extensions to see if they’re good enough. It may be that you just don’t need the flexibility of the MV2 code, and the new extension is fine.

You could switch to another browser; Mozilla claims they have no plans to deprecate MV2 in Firefox, and many of the extensions you use on Chrome are available for Firefox.

There are some forks of Chromium that have built in ad blocking (e.g. Brave) but some of these may come with other concerns.

There are also some commercial Enterprise browsers out there with their own ad-blocker engines which shouldn’t be impacted by this (I was involved in the evaluation of one, and it was pretty good, but I try not to recommend products!)

Vivaldi wants to keep MV2 functionality “for as long as it’s still available in Chromium”, which may be until June 2025.

Getting another year of life

Which brings us to another way of keeping going until June 2025. Google are granting Enterprises the ability to keep MV2 extensions by setting a management policy. We can make use of this even for unmanaged browsers!

If you go to about:policy in chrome and click on “Show policies with no value set” then you’ll see an entry called “ExtensionManifestV2Availability”. This can be used to control if MV2 can be used. The default value is “browser default” but we can set this to 2 to force it to be on.

How we do this will depend on your OS.

Obviously Enterprises using the Manager tool can set it there. But even without this the policy can be set on a machine by machine basis.

On Windows this is stored in the registry. We can set it from an admin command line:

C:\Windows\System32>reg add HKLM\Software\Policies\Google\Chrome /v ExtensionManifestV2Availability /t REG_DWORD /d 2
The operation completed successfully.

On Linux if you view the policy logs (“more actions / view logs”) you should see entries similar to

[VERBOSE] Skipping mandatory platform policies because no policy file was found at: /etc/opt/chrome/policies/managed
[VERBOSE] Skipping recommended platform policies because no policy file was found at: /etc/opt/chrome/policies/recommended

This tells us where the policy file needs to be placed. It may be different on Chromium. In my case I was then able to do

# mkdir -p /etc/opt/chrome/policies/managed
# echo '{
  "ExtensionManifestV2Availability": 2
}' > /etc/opt/chrome/policies/managed/v2.json

In both scenarios, if you go back to about://policy and click on “Reload policies” the value should show up.

With luck this will let us keep using MV2 extensions until June 2025.

This may come with a downside; Google are no longer accepting MV2 extensions into their store so if there is a security issue found in an extension then it may not get fixed. So definitely caveat emptor!

These changes are at the machine level, so every user of the machine will get this policy.

Enterprises could set the registry via a GPO and update the local file on Unix machines with their preferred management tool.

Home users can do this manually. It needs to be done on each machine, so if you have more than one machine then remember to make this change everywhere!

Summary

The MV2 deprecation is not popular. It may add some security benefits against rogue extensions but it definitely has downsides for the more privacy and security conscious folks.

In the long run we might see a resurgence of alternate browsers and the decline of Chromium based ones (much as Chrome killed Internet Explorer), but given many end users are likely just using Edge on Windows I won’t hold my breath.

On 9/11 deniers

Wed, 11 Sep 2024 15:49:31 -0400

For some reason this year a lot more 9/11 denialism has come across my social media feeds. I wonder if it’s because of the upcoming election.

And I just can’t…

I’d had enough a decade ago and wrote something then; I’m repurposing it here.

I was working on Wall Street the day it happened, just half a mile away. I’d only moved to the US 2 months earlier.

I spoke to people who were at WTC as it happened. I consoled a man who ran away rather than trying to help people, and thought this was a breach of his identity as a Christian.

When the towers collapsed the amount of dust caused a total black-out; you couldn’t see anything outside of the windows. Afterwards I could see a set of footprints in the dust, as if it was snow.

Eventually we were told to leave the island and I had to walk home (about 6 miles in massive hot sun). I was really impressed; on the other side of the Brooklyn Bridge all the office workers had brought out their water coolers and stuff and were handing out water to us as we walked past.

I still didn’t have any furniture in my apartment; a TV sitting on a stand, but no sofa or chairs; only an office chair I was using for the computer room (no tables for the computers either; they were sitting on their boxes). No mobile phone (I bought one the following weekend!) but I did have a landline that still worked.

Not my happiest of days.

Oddly, the next week I was scheduled for DSL install. I expected this to be canceled ‘cos all telco technicians would be working to fix Manhattan… but, no! They actually installed it a day early!

And then the denialism…

Please, just shut up.

There’s enough about that day and it’s consequences to be sorry about. Deranged rantings from loons does not help.

Whenever a “9/11 truther” intrudes on my presence I get angry at that idiot, where I would rather continue being sad for the pointless and useless loss of life; those lost on that day, those of the emergency services lost on that day (or as result of dust inhalation, etc), and those lost overseas as a result of the war.

All a waste of life. All pointless. All for nothing.

I remember the heroes of the day; those office workings giving us water, those who helped bolster spirits, and those who emergency workers who rushed into peril to help those in need.

I remember crying, I remember turning off the TV because I couldn’t handle it any more.

I remember talking with my parents back in England. I remember meeting my girlfriend the next evening and taking part in the vigil at Washington Square Park.

I remember the despair at seeing hate graffiti in Manhattan the day after. I remember the anger at the idiotic US government response. I remember my girlfriend walking in marches to stop the war (I didn’t march because I didn’t think it my place to march against the US government, since I wasn’t a US citizen).

I remember Fox News. I remember Fahrenheit 9/11.

I remember the parasites. I remember the stores opening to sell ‘personal protection’ against additional attacks; “office parachutes”, plastic to seal windows against air-born infection, personal bio-suits. (There was a store 2 blocks from the office)

What I don’t need is you.

Please… shut up.

Yubikey 5 is broken! Panic! Or not

Thu, 05 Sep 2024 11:34:04 -0400

I’d previously written about the Yubikey 5 and how we could use it to solve various use cases and when to trust it. Personally, I think it’s a great device for corporate authentication solutions.

But…

This week, Yubico released an advisory that stated that ECDSA private keys could be stolen from a Yubikey 5 that’s running firmware older than 5.7.0 (or 2.4.0 for the YubiHSM). This is due to a flaw in the cryptographic libraries written by Infineon and discovered by NinjaLab.

What is the issue?

There’s a lot of maths involved in the NinjaLab document, but basically it boils down to a certain action not being “time constant”. This means a successful action and a failed action can take different times. This timing can be observed, and this leads to a side channel attack. NinjaLab were able to do this and recover the ECDSA private key embedded into the device.

This is kinda bad; the promise of security keys is that the private key can never be leaked. Now we have no guarantees!

So time to panic?

(Gratuitous Hitch Hikers Guide reference)

Let’s look at the different authenticators on a yubikey and how they might be impacted

FIDO2

FIDO2 can work in two ways. When you register two a FIDO2 service a unique public/private key is created. The public key is sent to the server and this is then used to verify you when you login.

What changes is in how the private key is stored.

The first are known as “non-discoverable” credentials. Here the private key is encrypted with an embedded key in the yubikey and the encrypted key also sent to the server. When you attempt to login you are sent back the encrypted private key, the yubikey can then use the embedded key to decrypt this and then authentication happens as normal. This is nice because it means that you can register your device with unlimited services; no local storage is needed. However to login you need to tell the remote service who you are (typically requires a login) so it can send you the encrypted key.

The second is known as “discoverable” credentials. Here the private key is stored directly on the yubikey. Of course this takes up space so only a limited number can be stored. The advantage is that we don’t need to login to the remote service in order to get the private key, so it can be used for true passwordless logins.

You can see what discoverable credentials are on your machine by using the yubikey manager (you need to be Admin in Windows).

e.g using a test token from token2.com

C:\Program Files\Yubico\YubiKey Manager>ykman fido credentials list
Enter your PIN:
Credential ID  RP ID           Username                                        Display name                             
63cd48c8...    www.token2.com  token2_user_66d9dff6baa59_445105966d9dff6baa97  token2_user_66d9dff6baa59_445105966d9dff6baa97

C:\Program Files\Yubico\YubiKey Manager>

So there’s a potential difference in exposure between these credentials, and it’s up to the server to decide what type to use.

With a discoverable token then the attacker can see what sites are available for access and which may not need a password (‘cos true passwordless) but they will need the PIN. If they have the PIN then they could retrieve the private key for this site.

For non discoverable tokens the attacker could register your key to another site; they choose the username and password and a retrieve the master private key. They would still need the PIN, but this would allow them to attack multiple sites… if they knew what they were and your login details!

What about OTP, static secrets, PIV and PGP?

OTP doesn’t do this type of calculation. It’s based on a seed value and a generator function. This is not impacted.

Static secrets don’t do this either… but if someone has your yubikey then they can just press the button to get the secret. Don’t use static secrets!

Now, by default, PIV and PGP modes uses RSA keys for device attestation. These are not impacted. However a corporation could replace those attestation certs and so use ECDSA, but I’ve never thought this a good idea; the nice thing with using the inbuilt keys from Yubico is that you can use an off-the-shelf device straight from packaging and sent to an user.

PIV and PGP operations, however, could be done using ECDSA keys. PIV mode requires a PIN and PGP can be configured to use one. Operations on RSA keys aren’t impacted.

How easy is the attack?

Firstly, in order to do this attack you need access to the yubikey itself. This isn’t that big a deal; indeed it’s recommended that people have a backup key that’s stored somewhere secure in case they lose the first one. How often is that backup key checked? But physical access is needed.

NinjaLab claims they only need access to the key for a few minutes, but that doesn’t include the time needed to take apart the key, hook up into the internals and then reseal the key afterwards. Of course if the attacker doesn’t care if the key is noticed as missing…

You need pretty expensive equipment (maybe $11,000 worth according to ars Technica) to get the data. I could see this coming down in price.

For non-discoverable credentials you need to know what service the key has been registered to.

You may need a username and password to the service, and a PIN to access the credential.

So it’s not easy to do, but it’s not out of the realm of possibility. ars thinks “nation-state” level attackers but there may be more than that if the return is more than the cost.

Mitigations

I believe a PIN will help block attacks because it’s another thing the attacker needs to have in order to perform the side channel attack. As long as the PIN isn’t set to the default value, of course! Yubikeys only allow limited number of PIN retries before the application is locked and the secrets unreadable. PINs are harder to get than username/password information because the UI presented makes it harder to be phished. But it’s not impossible and keylogger malware would have access.

So definitely ensure any yubikeys in your environment are configured to require PINs and that the PIN isn’t set to the default. Note than “PIN” is a bit of a misnomer for yubikey; it can contain more than numbers, so allow alphanumerics in whatever app you provide for management of PIV/PGP certs.

Obviously use of RSA keys, rather than ECDSA, will block this attack.

Train staff to report missing keys immediately and have processes (SOC or even helpdesk) to revoke the token.

Require staff to verify they know where the backup key is (maybe force them to use it once a month?).

Wrap keys with tamper-evident seals with unique numbers (this will be annoying since it means off-the-shelf keys aren’t allowed, and it’s not enforceable, but it’d help) so that evidence of the key being taken apart is obvious.

Summary

It seems that the “big” risk is U2F/FIDO mode. Exposure could allow an attacker to authenticate as a user. But it’s not necessarily easy.

Do you need to worry about it? Or this is just yet another theoretical attack? That’s a risk management decision. Use of these “vulnerable” keys will still prevent most attacks on your users. It’s a targeted well financed attacker that’s the worry.

It may be that you would want to take a tiered approach; staff with access to highly sensitive data may need a new key with fixed firmware, but staff who just use it to access their laptop to read email don’t need an upgrade.

Hmm, I wonder how many vulnerable devices are still in the supply chain? I assume anything bought direct will have the new firmware, but what about if you buy from Amazon; will they still have old keys warehoused?

The main question I have, though, is where else this chipset and library has been used? What other devices may be vulnerable?

After all, it’s not the first time NinjaLab have broken hardware tokens

And, of course, remember 538

Crowdstrike issues

Fri, 19 Jul 2024 14:28:30 -0400

I was asked about today’s Crowdstrike issues on Windows.

Naturally I have some thoughts…

What went wrong? What I know.

Crowdstrike is an EDR (Endpoint Detection and Response) tool. (Well, they claim “XDR”, but that’s marketing). It has an agent component and a set of rule sets (called “channel file”). The agent has both user space and kernel space components to better give visibility into what is happening on the machine, and to be able to block bad things. It’s not an AV solution (“Oh, this is a known bad file; we’ll block it”) but it intercepts calls at runtime and determines if this is legitimate behaviour or not.

So Crowdstrike can act both as a visibility tool (it can tell you every process ever executed on your machines) and a defense tool (block bad stuff).

Now, in common with a lot of tools like this we need to keep both the agent and the channel files up to date; the agent needs to handle platform changes (eg a change in drivers or kernel structures) and the channel files need to be kept up to date to handle new threats and attacks.

Many organisations have requirements around roll out of new agents (e.g. test in a lab and then do a risk-based deployment; e.g. to dev then qa/cat and finally to prod) to minimise production outages.

In the case of Crowdstrike the channel files are automatically updated and this is where the problem occurred. Something in the channel files deployed at 04:09 UTC broke Windows causing it to BSOD.

Crowdstrike deployed a new set of files at 05:27 UTC and so any machine off network in that window should not have been impacted. Hopefully!

Complications

This isn’t a one-off BSOD; it also crashes the machine early in the reboot process; potentially too early for the agent to pick up the fixed update. (I’ve seen a tweet where someone claimed multiple reboots did resolve it through luck, but that was just a one-off). So, essentially, this will require human intervention on every affected machine.

Crowdstrike have provided instructions on how to fix and delete the bad file.

OK, that’s painful but it’s not impossible. Especially on servers.

More worrying could be end user devices (laptops, VDIs, etc) that are impacted. These instructions are not the sort of thing you’d expect your typical end user to perform; they may not even have the necessary knowledge (no local admin access). And yet more complication is that it’s more likely for end user devices will have Bitlocker on them! Devices in the office can have help desk people run around and do the work, but what about those working from home? Or salesmen in the field?

Because of when the bad files were pushed, We can hope that a lot of these machines were shutdown or in standby mode and so missed the update! Well, in EMEA and AMER regions, anyway. APAC will have been in the middle of the work day.

Haven’t we seen this before?

Yes! In 2019 a Symantec SEP update did almost exactly the same thing. AV tools have, in the past, also tried quarantining windows components because of bad signature files.

So we shouldn’t auto-download new rules?

Definitely agents should have a planned deployment. But what about the rules? This is where we need to look at the risks.

So what do up-to-date rules give us? Protection against the latest attacks; you may defended before you’re even aware of the threat because companies like Crowdstrike had lots of threat intelligence streams and see indicators of compromise and create rules to defend against them very early! It may not defend you against a novel zero-day attack, but it should prevent a day-1 attack.

The question you need to ask yourself is how many of these attacks have been defended against, and how much it would have cost to recover (dollars, reputation, lost current/future customers…). How quickly would a compromise spread? Do you have secondary controls that would minimise impact?

You do have a measure of that, right? You are monitoring your EDR logs to see what has been prevented; what are day-1? Many potential attacks may already be prevented by other tools (eg mail gateways blocking malware), so the day-1 risk might not be as high as it could be. But it’s not zero, otherwise we wouldn’t need tools like this!

You also need to look into how quickly your team would be able to validate new channel files. Are you able to perform a test suite daily? Or would new rules be delayed a week? How much can you automate this? Basically how big would your vulnerability window be.

EDITED TO ADD based on feedback elsewhere. We also need to consider cost of remediation. In this type of outage you have a level of comfort that no sensitive data was accessed. (Unless it’s a solarwinds type issue…). An exploit that wasn’t prevented due to having out of date rules has additional monetary cost (patching, investigation, regulatory reporting, law enforcement, etc) that may go on long after the event was detected.

These three factors (window, number of attacks prevented, cost) all weigh into the risk.

On the other side we have the risk of outages like this. Impact is massive, but since it seems almost everybody is impacted (yeah, an exaggeration, but not much of one) the chance of it causing you reputational impact and customer loss is a lot less.

What I do believe is that companies that provide rule sets like this (EDR, AV, etc) should provide the option to allow companies to decide for themselves. Perhaps they could run their own “day 0” and “day 1 deferred” channels; a company could have most machines on “day 1” with a set of test machines on “day 0”; if a test machine blows up they can hit the big red switch to halt production deployments.

So if we should keep doing updates, how do we mitigate this?

Traditional active-active or even warm-standby DR systems would have also picked up the new rules and died at the same time as production. Oops! So if your vendor product does have staggered “day 0” and “day 1” type channels then it’s worth thinking about having the prod/dr systems on different schedules. It may even be worth thinking about having DR ahead of production so if they die it won’t cause outages.

We also need to think about disk snapshots; could we roll back the machine to a state before it got the bad files? On a VM this is perfectly possible, but we need to be aware than any changes will be lost; audit logs, activity logs etc will be lost. These might be critical (regulatory) so get those logs off the machine ASAP (streaming to a remote store; put them on a NAS share; something…). Try to make what changes on a machine to be as small as possible.

Could we automate deployments? How quickly can you spin up new infrastructure and deploy your apps? If you’re in a modern shop with effective CI/CD pipelines this may already be part of your deployment process (especially in the cloud); recovery is “deploy new!”

Or, similarly, how much of the app needs to be deployed locally, vs stored on a NAS? If a new VM can be spun up and connects to a P: drive (P for aPP!) and auto-starts…

What about old-school cold-standby instances that just get sync’d once a day?

What about “dual boot” scenarios, similar to how appliances get handled? A TiVo (for example) has two OS partitions; an active one and an inactive. When an update comes in it gets written to the inactive partition, validated, and then the “active” flag gets switched. We can’t quite do this with Windows but possibly a second partition that’s sync’d from “active” on a nightly basis and a boot loader that will allow for booting from that second partition.

End users also need to be thought about; if we make their device as close to zero-permanence as possible (all data files stored on OneDrive or Google Drive or NAS; nothing on C:) then if that dies we can spin up new VDIs (even cloudy ones like Windows365 or Amazon Workspaces) and get the back working again. At least email and chat and common stuff…

I’m just spit-balling ideas here!

There will always be edge cases, but if we can standardize and make the endpoint as “irrelevant” as possible (basically just provide compute power) then recovery becomes quicker.

Summary

It’s easy to point fingers at Crowdstrike and say “you fucked up”. Well, they did! It will be interesting to see what went wrong, and what controls they put in place to prevent a similar occurrence. Other companies in the same areas (EDR, AV, etc) also need to learn these lessons so they don’t suffer the same problems in the future.

But as a customer you don’t want to be too nervous and change your processes. You need to evaluate the risk. It’s very possible this once-in-a-blue-moon event is less damaging than other courses of action.

Get the data, evaluate your environment, determine what mitigations you can put in place and then come up with a plan. Don’t just take the easy path of deferring rule updates for a day or two (or seven!) because that might expose you to greater risk than getting regular updates pushed live to prod!

Having the option would be nice, though.

Building a home router

Thu, 18 Jul 2024 14:52:43 -0400

WARNING: technical content ahead! There’s also a tonne of config files, which make this page look longer than it really is, but hopefully they’ll help other people who want to do similar work.

Back in 2017 I described how to build a home router based on CentOS 7.

C7 is now out of date, so I figured it was time to rebuild it, this time using Rocky Linux 9. That should give me another 8 years of security patches!

My requirements were mostly the same, but I wasn’t going to use WiFi on the machine; I’d already replaced that with Access Points. I also wasn’t so much with IPv6 any more; this is mostly because the Hurricane Electric tunnelbroker IP addresses were on a lot of “low reputation” lists, and I was getting annoyed with that. I still use IPv6 but only specific machines, with a static configuration, use it.

I figured it was also time to get a new machine. There are a LOT of cheap hand-sized computers, these days, including some dual port ones. I went for a CTone Mini PC; 8GB RAM, 256Gb SSD, dual 1Gbit NICs. Sounded perfect. Unfortunately it wasn’t. It uses RTL chipsets, which appear to be less performant than the existing machine with Intel NICs; it worked but speed test results were consistently 100Mbit/s slower. So I got a new SSD for my old machine and built fresh on that. If I failed to get it to work then I could just put the old SSD back in. The new slow machine isn’t wasted though; it’s now a backup just in case my primary ever fails.

Basic networking

Requirements

At home I have 3 VLANS; 10 (LAN), 11 (guestnet), 12 (IoT). Although I no longer need to (no wireless) I’m going to keep using bridges for everything and I’m going to keep using the old name.

So, to summarize, the result would look something like

Use	Bridge	Interface
Internet	br-wan	enp2s0
LAN	br-lan	enp3s0.10
Guest	br-guest	enp3s0.11
IoT	br-iot	enp3s0.12

For simplicity, I’m going to assign enp2s0 to $WAN, and enp3s0 to $LAN.

WAN

First we need to make sure there’s no existing connections defined from the OS install. This can be done with the nmcli con show command. If you see anything like Wired connection 1 then you should delete them with nmcli con del ....

You will lose network access doing this, so make sure you’re on the console!

Bridges are created with nmcli connection add type bridge ... and interfaces are associated with it with nmcli connection add type bridge-slave ... commands.

So the WAN interface would be created with

nmcli device set $WAN autoconnect yes

nmcli connection add type bridge con-name br-wan ifname br-wan bridge.stp no ipv4.method auto ipv4.dns "10.0.0.1 10.0.0.5" ipv4.dns-search "spuddy.org" ipv6.method disabled
nmcli connection add type bridge-slave con-name $WAN ifname $WAN master br-wan

In this I am hard-coding my internal DNS servers (this machine and my backup DNS server) and DNS search path.

Woohoo! At this point the machine should(!) have internet access.

Internal VLANs

Similarly we can define the 3 VLANs and bridges. br-lan will have a static IPv6 address, but the rest just have IPv4.

nmcli device set $LAN autoconnect no

nmcli connection add type bridge con-name br-lan ifname br-lan bridge.stp no ip4 10.0.0.1/24 ipv6.method manual ipv6.address 2001:470:1f07:dc4::1/64
nmcli connection add type vlan con-name $LAN.10 ifname $LAN.10 vlan.parent $LAN vlan.id 10 slave-type bridge master br-lan
nmcli device set $LAN.10 autoconnect yes

nmcli connection add type bridge con-name br-guest ifname br-guest bridge.stp no ip4 10.100.100.1/24 ipv6.method disabled
nmcli connection add type vlan con-name $LAN.11 ifname $LAN.11 vlan.parent $LAN vlan.id 11 slave-type bridge master br-guest
nmcli device set $LAN.11 autoconnect yes

nmcli connection add type bridge con-name br-iot ifname br-iot bridge.stp no ip4 10.100.200.1/24 ipv6.method disabled
nmcli connection add type vlan con-name $LAN.12 ifname $LAN.12 vlan.parent $LAN vlan.id 12 slave-type bridge master br-iot
nmcli device set $LAN.12 autoconnect yes

We can now see how this looks:

# nmcli dev
DEVICE     TYPE      STATE                   CONNECTION
br-wan     bridge    connected               br-wan
br-lan     bridge    connected               br-lan
br-guest   bridge    connected               br-guest
br-iot     bridge    connected               br-iot
enp2s0     ethernet  connected               enp2s0
enp3s0.10  vlan      connected               enp3s0.10
enp3s0.11  vlan      connected               enp3s0.11
enp3s0.12  vlan      connected               enp3s0.12
lo         loopback  connected (externally)  lo
enp3s0     ethernet  disconnected            --
enp1s0     ethernet  unavailable             --

# nmcli con
NAME              UUID                                  TYPE       DEVICE
br-wan            0102b304-effc-4437-a70a-aa3360d4d3e0  bridge     br-wan
br-lan            acf41256-e68b-40a8-a85a-944eb95035a6  bridge     br-lan
br-guest          a3c962bb-e605-46aa-8d43-7cb9db4ebad1  bridge     br-guest
br-iot            f7715bbd-1a7c-41fb-bfb0-5ddc2900048a  bridge     br-iot
enp2s0            e966376e-39e5-4789-aaaa-5a2c0154323d  ethernet   enp2s0
enp3s0.10         bfd85eca-989c-499c-89d2-cb4f57d0cde9  vlan       enp3s0.10
enp3s0.11         a63e60ce-e808-499e-bfb1-aedb5ee83541  vlan       enp3s0.11
enp3s0.12         2d775836-ab3b-4ccb-a120-c32e2ec0efc7  vlan       enp3s0.12
lo                f63cce44-1aed-467f-bd27-96348e58c8f2  loopback   lo

If we install the old bridge-utils package (from EPEL) we can also see very simply the bridge configurations

% brctl show
bridge name     bridge id               STP enabled     interfaces
br-guest        8000.000db9439cce       no              enp3s0.11
br-iot          8000.000db9439cce       no              enp3s0.12
br-lan          8000.000db9439cce       no              enp3s0.10
br-wan          8000.000db9439ccd       no              enp2s0

And IP addresses can be seen with ip -4 a. So far so good!

Config files.

NetworkManager is not meant to be manipulated via config files (I much prefer the old /etc/sysconfig/network-scripts) but there are files we can look at, in /etc/NetworkManager/system-connections. e.g.

% ls /etc/NetworkManager/system-connections/
br-guest.nmconnection  br-wan.nmconnection     enp3s0.11.nmconnection
br-iot.nmconnection    enp2s0.nmconnection     enp3s0.12.nmconnection
br-lan.nmconnection    enp3s0.10.nmconnection

IPv6

Hurricane Electric IPv6 tunnels are very easy to set up. You need three values; the tunnelbroker endpoint IPv4 address, and the two IPv6 address associated with each end. These will probably be ….::2 for your end, and ….::1 for their end.

nmcli connect add type ip-tunnel ifname he-sit mode sit remote $ipv4address ipv4.method disabled ipv6.method manual ipv6.address $connection::2/64 ipv6.gateway $connection::1

This will show up under nmcli con and nmcli dev

% nmcli dev | grep he-sit
he-sit     iptunnel  connected               ip-tunnel-he-sit

% nmcli con | grep he-sit
ip-tunnel-he-sit  6840bf43-8275-4ccc-88a1-6ff16caabea2  ip-tunnel  he-sit

The tunnel should come up (if your IPv4 address is correct on the HE end) and we can see this in ip -6 a output


24: he-sit@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1480 state UNKNOWN qlen 1000
    inet6 _connection_::2/64 scope global noprefixroute
       valid_lft forever preferred_lft forever

If all is going well, ping6 google.com should work!

PING google.com(lga25s41-in-x0e.1e100.net (2607:f8b0:4006:80f::200e)) 56 data bytes
64 bytes from lga25s41-in-x0e.1e100.net (2607:f8b0:4006:80f::200e): icmp_seq=1 ttl=119 time=9.21 ms
...

Now HE uses your IPv4 address to validate the tunnel is allowed to be established. If you’re on a dynamic IP then a simple script that calls their API would work.

PASS=YOUR_PASSWORD
USERID=YOUR_USERNAME
GTUNID=YOUR_TUNNEL_ID
HOST=https://$USERID:$PASS@ipv4.tunnelbroker.net

newip=$1

# Check to see what the end point currently is
x=$(wget -q --no-check-certificate -O - $HOST/tunnelInfo.php?tid=$GTUNID | sed -n 's/^.*<clientv4>\(.*\)<.*$/\1/p')

if [ "x$x" == "x$newip" ]
then
  echo Tunnel IP is already correct
  exit
fi

wget -q --no-check-certificate -O - "$HOST/nic/update?hostname=$GTUNID&myip=$newip"

Firewalls and routing

That’s the low level plumbing done. Now to make it look like a router

Forwarding

Routers need to forward packets… This is done by setting sysctl options. They can be made to persist by putting them into a config file; e.g.

% cat /etc/sysctl.d/forward.conf
net.ipv4.ip_forward = 1
net.ipv4.conf.all.forwarding = 1
net.ipv6.conf.all.forwarding = 1

At this point the 3 VLANs can see each other. Unrestricted and there’s no internet access, but we’re getting there!

firewalld

RL9 has two ways that can manipulate network table rules; the first is firewalld, and the second is to use nftables directly.

I prefer nftables, so we need to disable firewalld and enable `nftables

# systemctl disable --now firewalld
# systemctl mask firewalld
# systemctl enable --now nftables

nftables

I’m going to look at this similarly to how iptables did it. We have standard rules (INPUT/OUTPUT/FORWARD) we have to look at them from the perspective of the router. INPUT rules are those that impact packets that are targeted at the router; OUTPUT rules are those that originate from the router; FORWARD rules are those that pass through the router. This gets important for NAT where the packet targets the router’s external IP address. Similarly for NAT rules we have PREROUTING and POSTROUTING.

nftables has a bit more “fluff” needed to create the necessary structures, so that’s where we start

# Clear out all the rules
flush ruleset

# Create empty structures
add table ip filter
add table ip nat
add table ip6 filter

I’m also going to define some structures that we’ll use later.

add map nat fport { type inet_service : interval ipv4_addr . inet_service ; flag
s interval; }
add set ip nat reflect { type inet_service; }
add set ip nat this_host { type ipv4_addr; }

add set ip6 filter my_hosts { type ipv6_addr; }

IPv4

We can now start building the IPv4 rules. Let’s create the 3 standard chains and 2 more chains that will be used for logging of packets

add chain ip filter INPUT { type filter hook input priority 0; policy accept; }
add chain ip filter OUTPUT { type filter hook output priority 0; policy accept; }
add chain ip filter FORWARD { type filter hook forward priority 0; policy accept; }

# Log rules for input and forwarding
add chain ip filter LOGGER
add chain ip filter LOGGERF

The logging rules are pretty simple. We limit how many can be sent, to avoid DoSing ourselves with log messages!

# Logging rule for INPUT drops
add rule ip filter LOGGER limit rate 2/minute burst 5 packets counter log prefix "IPTables-Dropped: "
add rule ip filter LOGGER counter drop

# Logging rule for FORWARD drops
add rule ip filter LOGGERF limit rate 2/minute burst 5 packets counter log prefix "IPTables-Dropped-Forward: "
add rule ip filter LOGGERF counter drop

Now we can handle INPUT rules (remember, this is traffic meant for this machine, and it may come from any of the VLANs or internet). We want to allow all traffic from the LAN; guestnet and IoT have limited access. Everything else should be blocked. Remember, we’re only talking about traffic targeting the router; we’ll get to port forwarding later.

# Allow traffic for established connections, blocking invalid header format
add rule ip filter INPUT ct state related,established counter accept
add rule ip filter INPUT ct state invalid counter drop

# All localhost traffic allowed
add rule ip filter INPUT iifname "lo" counter accept

# The LAN can see this machine
add rule ip filter INPUT iifname "br-lan" counter accept

# Allow the world to ping this machine (amongst other things)
add rule ip filter INPUT ip protocol icmp counter accept

# Let guestnet and IoT see DNS servers on this machine (tcp+udp)
add rule ip filter INPUT iifname "br-guest" tcp dport 53 counter accept
add rule ip filter INPUT iifname "br-guest" udp dport 53 counter accept
add rule ip filter INPUT iifname "br-guest" tcp dport 853 counter accept
add rule ip filter INPUT iifname "br-guest" udp dport 853 counter accept

# IoT may also need NTP access
add rule ip filter INPUT iifname "br-iot" tcp dport 53 counter accept
add rule ip filter INPUT iifname "br-iot" udp dport 53 counter accept
add rule ip filter INPUT iifname "br-iot" udp dport 123 counter accept

# Log block traffic; this could be noisy, so only enable it if you
# want lots of logs!
## add rule ip filter INPUT counter jump LOGGER

# Block all other incoming traffic
add rule ip filter INPUT counter drop

OUTPUT rules are very simple. We allow everything, because this is just for traffic that originated on this machine.

add rule ip filter OUTPUT ct state related,established counter accept
add rule ip filter OUTPUT ct state invalid counter drop

Now we need to allow traffic from our internal networks to see the internet. I’m also going to add a special rule; if my mobile is on guestnet then it can see the whole of the WAN.

There are also some other special rules (eg allow guests to see my printer) that we need to handle

# Allow established connections to work, blocking invalid header format
add rule ip filter FORWARD ct state related,established counter accept
add rule ip filter FORWARD ct state invalid counter drop

# Allow the LAN to see the world
add rule ip filter FORWARD iifname "br-lan" counter accept

# Allow return traffic for NAT'd connections to work
add rule ip filter FORWARD iifname "br-wan" ct status dnat counter accept

# Allow my mobile devices to have access to everything
add rule ip filter FORWARD iifname "br-guest" ip saddr 10.100.100.8 counter accept

# Allow guestnet to see the printer
add rule ip filter FORWARD iifname "br-guest" ip daddr 10.0.0.10 counter accept

# Allow guestnet devices to see my MQTT server
add rule ip filter FORWARD iifname "br-guest" oifname "br-lan" tcp dport 1883 counter accept

# Allow guestnet to see the internet
add rule ip filter FORWARD iifname "br-guest" oifname "br-wan" counter accept

# Allow IoT devices to see my MQTT server
add rule ip filter FORWARD iifname "br-iot" oifname "br-lan" tcp dport 1883 counter accept

# Allow ESP8266 devices on IoT to be network updates
# (The process causes the ESP8266 to call back to a webserver on the host)
add rule ip filter FORWARD iifname "br-iot" oifname "br-lan" tcp dport 8266 counter accept

# Allow IoT devices to see the internet
add rule ip filter FORWARD iifname "br-iot" oifname "br-wan" counter accept

# Allow marked traffic (ie reflected traffic) to reach the destination
add rule ip filter FORWARD meta mark 100 counter accept

# Log and block all other forwarded traffic
add rule ip filter FORWARD counter jump LOGGERF
add rule ip filter FORWARD counter drop

NAT

OK! We have connectivity on the internal network, now we need to set up the NAT rules to allow the internal network to SNAT (source NAT) to reach the internet, and to allow the internet to DNAT (destination NAT) to allow it to reach specific internal services.

There’s also an odd edge-case. If you’re on the guest network and try to reach an internal service via the external IP address (e.g. a web server) then it all breaks. This is because the incoming traffic doesn’t arrive via the br-wan interface and so isn’t NAT’d. In iptables days we used a set of “reflection” rules (see the earlier blog entry about this). With nftables we can be smarter. We can set up some rules that use sets and maps to hold the information.

So to define incoming port connections and those we want reflected:

# Port forwarding for incoming traffic

add element nat fport { 80 : 10.0.0.100 . 80 }   # http
add element nat fport { 443 : 10.0.0.100 . 443 } # https
add element nat fport { 22 : 10.0.0.110 . 22 }   # ssh
add element nat fport { 6881-6999 : 10.0.0.120 . 6881-6999 } # RL9 torrents

# These ports should be reflected.  Basically the ports from fport that
# people might want to access
add element nat reflect { 80, 443, 22 }

We’re now in a position to define a set of rules

add chain ip nat INPUT { type nat hook input priority 100; policy accept; }
add chain ip nat OUTPUT { type nat hook output priority -100; policy accept; }
add chain ip nat PREROUTING { type nat hook prerouting priority -100; policy accept; }
add chain ip nat POSTROUTING { type nat hook postrouting priority 100; policy accept; }

# Log specific incoming traffic
add rule ip nat PREROUTING iifname "br-wan" tcp dport 22 limit rate 2/minute burst 5 packets counter log prefix "IPTables-New-SSH: "

# Tag internal reflection traffic
add rule ip nat PREROUTING ip saddr 10.0.0.0/8 ip daddr @this_host tcp dport @reflect mark set 100

# DNAT incoming traffic from the internet (or internal send to br-wan address)
add rule ip nat PREROUTING ip daddr @this_host ip protocol tcp dnat ip  addr . port to tcp dport map @fport

# Do NAT on egress traffic to the internet
add rule ip nat POSTROUTING oifname "br-wan" counter masquerade

# Also NAT marked traffic from internal to external IP
add rule ip nat POSTROUTING meta mark 100 counter masquerade

@this_host

You might have noticed the @this_host rules. This refers to the set we defined earlier… but we haven’t populated it, yet!

There are two rules that work by have a PREROUTING rule detect traffic designed for reflection, and this sets a mark on it; the POSTROUTING rule detects this mark and ensures the traffic is masqueraded (SNAT’d).

The other rule using @this_host is the general incoming forwarding rule, which uses the fport map we filled out earlier. This catches both internet and VLAN initiated traffic that’s trying to hit this machine and ensures they’re both forwarded (DNAT’d) to the right target.

However, this set needs to hold the external IP address of the machine. We can set it in two places.

dhcp hooks

Whenever the br-wan interface gets assigned an address from DHCP we can arrange for a script to be called. This can simply populate the external IP address (passed as a variable) into the set:

% cat /etc/NetworkManager/dispatcher.d/nft_host
#!/bin/sh

if [ "$1" == "br-wan" -a -n "$DHCP4_IP_ADDRESS" ]
then
  nft "flush set nat this_host;add element nat this_host { $DHCP4_IP_ADDRESS }"
  echo `date`: Put $DHCP4_IP_ADDRESS into set
fi

cron job

Just in case the hook doesn’t fire properly, or if you reload the rules (and so have an empty set) we can also run a cron job every 5 minutes that can check the br-wan IP address, and if it’s not in the set we add it. I call this script reflect.

#!/bin/sh

export PATH=/sbin:$PATH

export DHCP4_IP_ADDRESS=$(ip -4 address show dev br-wan | sed -n 's/ *inet \([0-9.][0-9.]*\)\/.*/\1/p')

# Check to see if this is in the NFT ruleset
if [ -z "$(nft list set ip nat this_host | grep "{ $DHCP4_IP_ADDRESS }")" ]
then
  /etc/NetworkManager/dispatcher.d/nft_host br-wan dhcp4-change
fi

IPv6

Phew, that was a lot of work. IPv6 is a lot simpler; we allow any machine on the LAN to see the internet on IPv6 and we only allow a limited subset of external machines (machines I own) to be able to reach in from the internet.

We can specify the trusted machines by putting them in the my_hosts set

# My IPv6 hosts

add element ip6 filter my_hosts { \
  XXXX:XXXX:XXXX:XXXX:1, \
  YYYY:YYYY::YYYY:YYYY:YYYY:YYYY, \
  ZZZZ:ZZZ:ZZ:ZZ:ZZZZ:ZZZZ:ZZZZ:0 \
}

And now the IPv6 rules are pretty simple:

add chain ip6 filter INPUT { type filter hook input priority 0; policy accept; }
add chain ip6 filter OUTPUT { type filter hook output priority 0; policy accept;
 }
add chain ip6 filter FORWARD { type filter hook forward priority 0; policy accept; }

add rule ip6 filter INPUT ct state related,established counter accept
add rule ip6 filter INPUT ct state invalid counter drop
add rule ip6 filter INPUT iifname "lo" counter accept
add rule ip6 filter INPUT iifname "br-lan" counter accept

add rule ip6 filter INPUT meta l4proto ipv6-icmp counter accept

add rule ip6 filter INPUT ip6 saddr @my_hosts counter accept

add rule ip6 filter INPUT counter drop

add rule ip6 filter OUTPUT ct state related,established counter accept
add rule ip6 filter OUTPUT ct state invalid counter drop

add rule ip6 filter FORWARD ct state related,established counter accept
add rule ip6 filter FORWARD ct state invalid counter drop
add rule ip6 filter FORWARD iifname "br-lan" counter accept

add rule ip6 filter FORWARD meta l4proto ipv6-icmp icmpv6 type destination-unreachable counter accept
add rule ip6 filter FORWARD meta l4proto ipv6-icmp icmpv6 type packet-too-big counter accept
add rule ip6 filter FORWARD meta l4proto ipv6-icmp icmpv6 type time-exceeded counter accept
add rule ip6 filter FORWARD meta l4proto ipv6-icmp icmpv6 type parameter-problem counter accept

add rule ip6 filter FORWARD ip6 saddr @my_hosts counter accept

add rule ip6 filter FORWARD counter drop

Get this to load on reboot

What I do is have a complete list of all my rules in a file; I’ll include it below. We can then load this with the single command nft -f <filename>

Earlier we ran systemctl enable --now nftables. This picks up /etc/sysconfig/nftables.conf so we can now save the new rules

nft list ruleset > /etc/sysconfig/nftables.conf

Of course after reloading the rules, @this_host is empty so we also need to run the reflect command (or wait for the cron job).

We can do all of this in one line

nft -f nft.rules ; nft list ruleset >| /etc/sysconfig/nftables.conf ; reflect

The final nft ruleset

# Load this rules with
#   nft -f <this file>
# Save the rules so they apply on reboot with
#   nft list ruleset > /etc/sysconfig/nftables.conf
#
# In one command
#
#   nft -f nft ; nft list ruleset >| /etc/sysconfig/nftables.conf ; reflect
#
# Remember to configure nftables to start at boot time, and to disable
# firewalld
#
# systemctl disable --now firewalld
# systemctl mask firewalld
# systemctl enable --now nftables

###############################################################################

# Clear out all the rules
flush ruleset

# Create empty structures
add table ip filter
add table ip nat
add table ip6 filter

add map nat fport { type inet_service : interval ipv4_addr . inet_service ; flags interval; }
add set ip nat reflect { type inet_service; }
add set ip nat this_host { type ipv4_addr; }

add set ip6 filter my_hosts { type ipv6_addr; }

###############################################################################

# This should be the only configuration needed

# Port forwarding for incoming traffic

add element nat fport { 80 : 10.0.0.100 . 80 }   # http
add element nat fport { 443 : 10.0.0.100 . 443 } # https
add element nat fport { 22 : 10.0.0.110 . 22 }   # ssh
add element nat fport { 6881-6999 : 10.0.0.120 . 6881-6999 } # RL9 torrents

# These ports should be reflected.  Basically the ports from fport that
# people might want to access
add element nat reflect { 80, 443, 22 }

# My IPv6 hosts

add element ip6 filter my_hosts { \
  XXXX:XXXX:XXXX:XXXX:1, \
  YYYY:YYYY::YYYY:YYYY:YYYY:YYYY, \
  ZZZZ:ZZZ:ZZ:ZZ:ZZZZ:ZZZZ:ZZZZ:0 \
}

###############################################################################

# This ruleset is split into two parts; ip and ip6

###############################################################################
#
# IPv4
#
###############################################################################

# Basic rules:
#   LAN can see everything
#   Guestnet and IoT network can see a few internal resources (eg DNS)
#     and the internet
#   Outside world can see a few exposed services


# Define the 3 standard filters:
#     INPUT: Traffic targeting this machine
#    OUTPUT: Traffic originating from this machine
#   FORWARD: Traffic goin through this machine

add chain ip filter INPUT { type filter hook input priority 0; policy accept; }
add chain ip filter OUTPUT { type filter hook output priority 0; policy accept; }
add chain ip filter FORWARD { type filter hook forward priority 0; policy accept; }

# Log rules for input and forwarding
add chain ip filter LOGGER
add chain ip filter LOGGERF

######
###### Logging rules
######

# Logging rule for INPUT drops
add rule ip filter LOGGER limit rate 2/minute burst 5 packets counter log prefix "IPTables-Dropped: "
add rule ip filter LOGGER counter drop

# Logging rule for FORWARD drops
add rule ip filter LOGGERF limit rate 2/minute burst 5 packets counter log prefix "IPTables-Dropped-Forward: "
add rule ip filter LOGGERF counter drop

######
###### INPUT rules
######

# Allow traffic for established connections, blocking invalid header format
add rule ip filter INPUT ct state related,established counter accept
add rule ip filter INPUT ct state invalid counter drop

# All localhost traffic allowed
add rule ip filter INPUT iifname "lo" counter accept

# The LAN can see this machine
add rule ip filter INPUT iifname "br-lan" counter accept

# Allow the world to ping this machine (amongst other things)
add rule ip filter INPUT ip protocol icmp counter accept

# Let guestnet and IoT see DNS servers on this machine (tcp+udp)
add rule ip filter INPUT iifname "br-guest" tcp dport 53 counter accept
add rule ip filter INPUT iifname "br-guest" udp dport 53 counter accept
add rule ip filter INPUT iifname "br-guest" tcp dport 853 counter accept
add rule ip filter INPUT iifname "br-guest" udp dport 853 counter accept

add rule ip filter INPUT iifname "br-iot" tcp dport 53 counter accept
add rule ip filter INPUT iifname "br-iot" udp dport 53 counter accept
add rule ip filter INPUT iifname "br-iot" udp dport 123 counter accept

# Log block traffic; this could be noisy, so only enable it if you
# want lots of logs!
## add rule ip filter INPUT counter jump LOGGER

# Block all other incoming traffic
add rule ip filter INPUT counter drop

######
###### OUTPUT rules
######

# Output is easy; we let this machine talk to anything
add rule ip filter OUTPUT ct state related,established counter accept
add rule ip filter OUTPUT ct state invalid counter drop

######
###### FORWARD rules
######

# Allow established connections to work, blocking invalid header format
add rule ip filter FORWARD ct state related,established counter accept
add rule ip filter FORWARD ct state invalid counter drop

# Allow the LAN to see the world
add rule ip filter FORWARD iifname "br-lan" counter accept

# Allow the VPN to see everything
add rule ip filter FORWARD iifname "tun0" counter accept

# Allow return traffic for NAT'd connections to work
add rule ip filter FORWARD iifname "br-wan" ct status dnat counter accept

# Allow guestnet to see external DNS servers
add rule ip filter FORWARD iifname "br-guest" tcp dport 53 counter accept
add rule ip filter FORWARD iifname "br-guest" udp dport 53 counter accept
add rule ip filter FORWARD iifname "br-guest" tcp dport 853 counter accept
add rule ip filter FORWARD iifname "br-guest" udp dport 853 counter accept

add rule ip filter FORWARD iifname "br-iot" tcp dport 53 counter accept
add rule ip filter FORWARD iifname "br-iot" udp dport 53 counter accept

# Allow my mobile devices to have access to everything
add rule ip filter FORWARD iifname "br-guest" ip saddr 10.100.100.8 counter accept

# Allow guestnet to see the printer
add rule ip filter FORWARD iifname "br-guest" ip daddr 10.0.0.10 counter accept

# Allow guestnet devices to see my MQTT server
add rule ip filter FORWARD iifname "br-guest" oifname "br-lan" tcp dport 1883 counter accept

# Allow guestnet to see the internet
add rule ip filter FORWARD iifname "br-guest" oifname "br-wan" counter accept

# Allow IoT devices to see my MQTT server
add rule ip filter FORWARD iifname "br-iot" oifname "br-lan" tcp dport 1883 counter accept

# Allow ESP8266 devices on IoT to be network updates
# (The process causes the ESP8266 to call back to a webserver on the host)
add rule ip filter FORWARD iifname "br-iot" oifname "br-lan" tcp dport 8266 counter accept

# Allow IoT devices to see the internet
add rule ip filter FORWARD iifname "br-iot" oifname "br-wan" counter accept

# Allow marked traffic (ie reflected traffic) to reach the destination
add rule ip filter FORWARD meta mark 100 counter accept

# Log and block all other forwarded traffic
add rule ip filter FORWARD counter jump LOGGERF
add rule ip filter FORWARD counter drop

######
###### NAT rules
######

add chain ip nat INPUT { type nat hook input priority 100; policy accept; }
add chain ip nat OUTPUT { type nat hook output priority -100; policy accept; }
add chain ip nat PREROUTING { type nat hook prerouting priority -100; policy accept; }
add chain ip nat POSTROUTING { type nat hook postrouting priority 100; policy accept; }

# Log specific incoming traffic
add rule ip nat PREROUTING iifname "br-wan" tcp dport 22 limit rate 2/minute burst 5 packets counter log prefix "IPTables-New-SSH: "

# Tag internal reflection traffic
add rule ip nat PREROUTING ip saddr 10.0.0.0/8 ip daddr @this_host tcp dport @reflect mark set 100

# DNAT incoming traffic from the internet (or internal send to br-wan address)
add rule ip nat PREROUTING ip daddr @this_host ip protocol tcp dnat ip  addr . port to tcp dport map @fport

# Do NAT on egress traffic to the internet
add rule ip nat POSTROUTING oifname "br-wan" counter masquerade

# Also NAT marked traffic from internal to external IP
add rule ip nat POSTROUTING meta mark 100 counter masquerade

###############################################################################
#
# IPv6
#
###############################################################################

# IPv6 is simpler; there's no NAT involved.  We allow LAN to see the
# world and only allow trusted external machines to see in to LAN

add chain ip6 filter INPUT { type filter hook input priority 0; policy accept; }
add chain ip6 filter OUTPUT { type filter hook output priority 0; policy accept; }
add chain ip6 filter FORWARD { type filter hook forward priority 0; policy accept; }

add rule ip6 filter INPUT ct state related,established counter accept
add rule ip6 filter INPUT ct state invalid counter drop
add rule ip6 filter INPUT iifname "lo" counter accept
add rule ip6 filter INPUT iifname "br-lan" counter accept

add rule ip6 filter INPUT meta l4proto ipv6-icmp counter accept

add rule ip6 filter INPUT ip6 saddr @my_hosts counter accept

add rule ip6 filter INPUT counter drop

add rule ip6 filter OUTPUT ct state related,established counter accept
add rule ip6 filter OUTPUT ct state invalid counter drop

add rule ip6 filter FORWARD ct state related,established counter accept
add rule ip6 filter FORWARD ct state invalid counter drop
add rule ip6 filter FORWARD iifname "br-lan" counter accept

add rule ip6 filter FORWARD meta l4proto ipv6-icmp icmpv6 type destination-unreachable counter accept
add rule ip6 filter FORWARD meta l4proto ipv6-icmp icmpv6 type packet-too-big counter accept
add rule ip6 filter FORWARD meta l4proto ipv6-icmp icmpv6 type time-exceeded counter accept
add rule ip6 filter FORWARD meta l4proto ipv6-icmp icmpv6 type parameter-problem counter accept

add rule ip6 filter FORWARD ip6 saddr @my_hosts counter accept

add rule ip6 filter FORWARD counter drop

Work/Life Balance

Thu, 11 Jul 2024 08:32:26 -0400

This isn’t my normal tech-ish posting; this is a more personal view at how Corporate America and tech startups and the like are abusing their workforce. I don’t mean the sort of abuse seen in the service industry (below minimum wages needing to be supplemented with tips; excessive overtime; all that stuff). I’m talking about white collar tech jobs. The sort of jobs I did; likely the sort of jobs you’re doing (if you’re reading this blog); office workers…

Quiet quitting

The phrase “quiet quitting” appeared a lot in the press over the past few years; it’s probably peaked in public consciousness but is now there as a back ground thought. It colours attitudes; “Oh Harry is leaving at 5 on the dot every day…”

The thing is, this isn’t new. It’s not a Gen-Z or a Millennial thing. I’m at the older edge of Gen-X and I’d been “quiet quitting” for a couple of decades.

I started my first job in July 1990. I enjoyed what I was doing and I was learning a lot. I was still mostly a 9-5 in the office (just because that was expected commute times) but I would sometimes stay late. I remember one Sunday, around 1:30pm, just as we were at the table for our Sunday Dinner when the phone rang; one of my users was having trouble dialing into the computer so he couldn’t work (time critical). I was able to talk him through the alternate connection; he was happy. I was happy to have helped. Everybody won.

My second job was a bit like that; again mostly 9-5 but one weekend a server had died (it was a known Solaris kernel bug) so I went into the office (a 1.5hr commute) to restart it. Another weekend (the Saturday before Christmas) the firewall died, so again I went in to fix it. And there was the long Easter 4-day weekend I spent upgrading ‘Orrible Financials. Ugh.

Basically I would go “above and beyond” in emergency situations, but my daily work was mostly a 9-5.

When I moved to America my attitude changed a little; I was no longer in an operations role, so it was less likely for me to have to do emergency work. When I was “level 4” (with good L2 and L3 teams running my stuff) only my boss had my cellphone number; only he was allowed to contact me. My 8:20-5 (‘cos of train times) slowly became a 8:20-4:45 (so I could catch the 5:30 train).

Yes, I still did “emergency” stuff (e.g. the early 2010s JPMC data breach had me working late hours analysing audit logs, building up a profile of activity performed by the attackers, demonstrating where controls had blocked them and where we were blind) but my “BAU” activity was strictly time bound.

In my last job, before COVID, it was closer to 8:35-4:20 (so I could catch the 4:55 train). Even when I worked from home I was strict 8:20-4:20. I resented people creating 8am meetings (and frequently missed them) and 4-5 meetings. Nope.

I was essentially “quiet quitting” but the work got done…

Corporate mis-incentives

And this is where modern corporate behaviour has made the type of working hours I did become important enough to flag up and cast it as a moral failing of the workers.

For the past decade or more the motto of Corporate America appears to be “Do More With Less”. Workers are encouraged to do more work (oh, but don’t work harder; work smarter!) with less resources and less time. That person now working 9-5 and not willing to go above and beyond… that person is dragging the company down. Why can’t you just stay that extra half hour? Don’t you care about the company?

NO.

I don’t care. The company is not my family; it’s not my friend. I have a commercial agreement with the company; “work these hours, do your job, get paid $$$“. You want to alter the deal? It goes both ways; you’re not Darth Vader, you don’t get to do this.

And this is where the mis-incentive’s start to show. I frequently hear “I need you to work more hours and log that time on your time sheets so I can show to senior management the need for extra headcount”.

The argument goes “if all my people are working at 125% then I need 25% more people”. But it doesn’t work that way; if you’re working at 125% the company will happily defer the “get more people” part because that impacts the bottom line. And defer it. And defer it. And suddenly 9-6 has become the new norm.

Worse, people get RIF’d (Reduction In Force; basically their job is terminated to reduce costs). And people who voluntarily leave don’t have their positions opened for a replacement. It’s pretty much a one-way-ratchet; resources get tighter and tighter.

Because the company is measuring “work completed” and seeing the numbers look good they have no reason to add more resources.

A better measure should be the backlog of work that doesn’t get done. Everyone works their 9-5 and incomplete work is incomplete.

Now the story goes, “My staff are working 100% but there’s work that isn’t getting done because there’s too much to do. I need more staff.” The company has incentive to get those people because the work not being done is costing and hurting the bottom line (or is declared not important and so not needing to be done).

But, instead, they claim “These young kids don’t want to do hard work any more! They’re lazy! Quiet quitting is the new scourge!” all in an attempt to guilt you to overwork yourself for their dollar.

Work to live

A number of years ago I was on a quarterly all hands meeting, and the big boss was asked how he felt about work-life balance. The answer disgusted me; basically it went something like “Yes, I believe in work-life balance but if you’re on vacation and not keeping up with your emails then you’re not doing your job. I always read my emails”.

The vast majority of us need a paycheck in order to live. In America you also need it to obtain health care (a rant for another day). But that’s where it should end. For most of us, your job is not your life. You need to be able to switch off at the end of the work day and be you, not corporate drone 1275567. The company doesn’t care about you as “you”, they just care about you as that drone. All those “health programs” they run are for their bottom line (fewer people using the health plan results in lower costs) even if they sell it as a “we care” benefit.

Aside: HR isn’t your friend; they’re there to protect the company from legal issues.

Quiet quitting shouldn’t even be a concept. Instead we should be framing this in terms of rejecting corporate abuse; of rejecting the “work harder, peon” mentality.

We’re first and foremost human beings (“Hi to you future AIs reading this!”) and need to treat ourselves as such.

(Of course if you pick a job where you get 6 figure bonuses because you’re expected to be in crunch mode 24x7 then that’s the choice you made; most of us didn’t choose that life!)

Summary

It’s easy for me to say “quiet quitting is good” (especially since I’m now retired!) because my job was never really at risk. I always got my work done; my boss was always happy; I got “exceeds” ratings.

Not everyone is in this position, though. Some people are desperate to keep their job and would struggle without it (especially in America where so many things are tied to work). So I’m not going to tell anyone not to work that 125%. I am hoping that you realise what is being done to you and, perhaps, be on the lookout for another job which might get you out of the abusive relationship with your employer (yeah yeah “better the Devil you know”).

And, may I whisper it in your ear, consider a union for your work space? The pendulum has swung back so far that union protections may be warranted!

API Security at the gateway

Wed, 12 Jun 2024 09:07:22 -0400

When it comes to talking about API Security there are many facets and paths the conversation can take.

We might want to talk about from an AppDev security perspective; after all, an API is just code, so your SAST/DAST type processes apply.

We might want to talk about it in terms of authentication; after all, you need credentials to access an API and there’s many different ways this can be done (Basic Auth, mutual TLS, Oauth, HMAC…); this would also include when anonymous APIs are OK!

We might want to talk about it in encryption terms; is TLS enough or should we do payload or field level encryption?

We might…

I want to talk about it in terms of the gateway.

WAFs

Normally a WebApp needs to be behind a WAF in order to provide some protection in depth; the WAF can block abusive behaviour before it even enters your network, so common attacks (SQL injection, volumetric, buffer overflow, etc) can easily be prevents.

Side note: This doesn’t mean your app never needs worry about this; WAFs can be bypassed. This is just another layer of protection!

In a large organisation we typically find WAFs become centralised and under control of a specialist team. This adds some friction to the development process, but there’s a lot of benefits; e.g. this team has more experience in understanding attacks and can craft rules to block new ones; the WAFs can alert to the SOC; centralised logging can assist in forensics… Also it’s important to note that not all WAFs protect as well; if an app team spins up their own (eg AWS WAF, or even Apache with modsecurity) we have the management problem (how do we ensure the latest rules are deployed; how to we get visibility/alerts to the SOC; how do we even know the AppDev team deployed properly? And is this WAF providing the same level of protection that the centrally managed ones provide?).

The problem with WAFs is that they are more focused on WebApp type traffic. APIs work differently; yes, they’re still https requests and so suffer some of the same problems (SQL injection, overflow) but there’s also new challenges; e.g this is highlighted by the OWASP API Top 10 being different to the standard OWASP Top 10.

WAAP

In recognition of this, some WAF vendors have tried to introduction some level of API protection into the same product, resulting in “Web Application and API Protection” (WAAP) products. Different vendors have different levels of maturity when it comes to API Security, and few actually describe what they actually do. It can sometimes be a marketing term, just so they can tick a box on a Gartner score card.

If you already have a WAF vendor that claims they can do WAAP then definitely look at the functionality; it may be sufficient for your needs.

Security at the API Gateway

Many APIs go through a gateway, which can do some sanity checking, routing of requests to different backend systems, enforce authentication requirements, and so on. Since these gateways act as an ingress point into your application (and thus your network and environment) this is another place where it potentially makes sense to centralize.

And with a centralised API Gateway comes the ability to add security controls and make it a Policy Enforcement Point (PEP). Advanced API Gateways even have data flows that can be hooked into and process flows defined.

There’s a number of benefits to providing API Security at this layer; all the “centralised WAF” arguments apply but, in addition, we’re already doing TLS termination here so we can do inspection and enforcement without adding any noticeable additional latency to the request (an external WAF or WAAP means another TLS decrypt, resend which will add delays). AppDevs seem to care about the performance of their APIs, for some reason, so now we have the ability to add security without making them cry.

What should an API Security Gateway offer?

Inventory

Since the solution sees all the traffic (request, response) it can automatically build up a list of endpoints being called and the types of data each endpoint handles. It would be able to classify each and flag those that handle sensitive information (“Oh, this handles PCI data”, “that’s an SSN”, “Name and address seen!”).

A solution could also be able to compare this discovered inventory to the published specs (eg OpenAPI) and determine if they match or if you have endpoints that aren’t part of the published spec, or you publish endpoints that don’t actually exist.

It could also track changes in the inventory over time and alert on important changes (“Huh, this endpoint is now returning SSNs where it didn’t before”).

Another useful output would be related to the performance of each endpoint; the system has timings for how long each request/response took. This could potentially help developers detect bottlenecks (wow, a security tool helping developers? Unheard of!).

Attack detection

This is where an API Specific solution can distinguish itself from a WAF. APIs tend to be called in a pattern of behaviour; an app would call A, B, C, D in order. The solution could learn this. If it then sees traffic going A, D then we have an anomaly.

Because the solution sees traffic it could also detect when endpoints are being abused (eg if the response size goes up massively in volume, or if the number of requests to an endpoint increases; seeing authentication tokens being used from an unexpected source; excessive “fail” responses potentially indicating an enumeration attack).

Many of the OWASP API Top 10 issues can be detected just by analysing the traffic.

Of course it also should do all the standard WAF-like stuff (SQL injection, buffer overflow, bot detection, volumetric attacks etc etc).

Alerting

This should go without saying; what’s the point of having all this detection if it can’t alert the SOC, or call a SOAR, or even just log high fidelity events to a SIEM.

Blocking

Once these alerts and logs have been validated and a level of confidence in the results have been achieved then the system should be able to automatically block bad traffic.

This blocking needs to be smart, though. Blocking by IP address is kinda pointless; attackers use botnets and change source IPs on a regular basis. Blocking one IP will just result in a new one starting up a short while later. This is lesson learned from WAF management.

Instead the blocking needs to be done on patterns; is there something about the requests that can be used? Is there something about the sequence of requests that can be used?

This is likely to be biggest differentiator between products; if the vendor just says “yes, we can automatically configure the firewall to block IP addresses” then you might want to look elsewhere.

DAST

Since the system has discovered API endpoints and potentially has a Spec file, it has a lot of information that can be used to generate automated testing of the endpoints. It can call them with known bad data; test against the OWASP issues (BOLA - Broken Object Level Authorizations - being a common one).

Results of these tests could be fed into existing vulnerability management processes and so tracked to remediation.

What characteristics do we need?

Speed

I’ve already mentioned that developers will cry if we slow down their apps, so a lot of the processing for all the above should be offloaded from the gateway.

Ideally part of the “process flow” should fork off an asynchronous copy of the data and response and send it to a processing server. In this way the original data flow is barely impacted.

Obviously in an enterprise environment you’d have multiple servers behind a load balancer for performance and resiliency. It also makes sense to put the processing server network-close to the gateway, especially since some of the “should this be blocked” logic could be run on this server and the gateway just makes a call-out to it.

Redaction

This is important, especially if (as in the above picture) we’re sending data to the cloud. We don’t want to send PCI scoped data to the cloud (unless you want to include that as part of your CDE). This is where the processing server also helps; it can perform the analysis and then redact or hash the data before it’s sent upstream.

The upstream systems will need to have something to correlate on, so the redaction process needs to be consistent (e.g. same credit card number always get hashed to a consistent value) but the raw data itself need not be sent.

Configuration promotion

No matter how good the auto-detection of sensitive data is, the system may make mistakes (eg miss one thing, or incorrectly classify an element). The system needs to allow for custom overrides.

Now for safety the developers should be able to verify and set these overrides when developing the code or testing in QA. After all, real data shouldn’t be used here (you don’t use real data in DEV…).

Ideally we’d then want an automated way of promoting the resulting configuration and classification of fields to production before the new code gets promoted. Otherwise there’s a risk of sensitive data being exposed by mistake.

Multi-team

In a large enterprise you may have multiple development teams working on different code, potentially in totally different lines of business. And there’s no reason why the Card Issuer team should be able to see the details (or even change the classification configuration) for the Investment Banking trading APIs.

Your AppSec team has possibly already solved this issue (e.g. only allowing teams to see their own SAST/DAST results) so being able to re-use their solution (Active Directory groups… it’s nearly always AD groups!) would be nice.

API Driven

It’s all well and good having pretty UIs, but management of the platform itself needs to be automatable. We might want to automate the association of developer team to endpoints, we might want to automate updated classifications, or override blocking rules, or…

The system must be API driven, with full documentation.

Summary

There’s a lot here and it covers a lot of areas. Some parts of it impact the AppSec team (DAST, vulnerability management, code promotion, automation), other parts impact the network security team (attack detection and blocking), other parts impact the development team (seeing their inventory, allowing them to classify endpoints, performance metrics). And, of course the SOC and related teams are also impacted.

Aside: We’re going to see more of this; the “siloed” approach to security is going away as solutions cross more and more boundaries.

I feel a solution like this is going to be critical; API attacks are extremely common as more and more applications become API driven (WebApp, mobile apps, b2b tools…). The API is what exposes your data. We need a WAF-like solution!

Imposter Syndrome

Tue, 04 Jun 2024 10:40:19 -0400

We all know what imposter syndrome is. We may all have suffered from it at some point. I know I did.

We may even know, rationally, that this isn’t a sensible thing. One good representation of this was from David Whittaker

Yet despite this whenever I started a new job I was always worried that I wasn’t the right person for it; that I’d fail to deliver. I feared messing up and getting called out as being incompetent.

As I grew into the role and started to deliver products and saw how my work improved the company my confidence would grow and I would be more comfortable; “OK, I can do this job.”

Soft deliverables

This is great when there’s hard deliverables; something you can point to and say “I did this”. When you see people using the stuff you create. When you overhear people talking to each other about it.

But what do you do when your deliverables are less concrete? When you don’t build but influence, or direct; when you build strategy; when people ask you for advice and your answers can affect a multi-million dollar project.

That’s the situation I found myself in when I joined my last job. I’d moved from an engineering role to an Enterprise IT Architect role.

Now a good engineer (and I like to think I’m a good engineer) must also have architecture skills. The decisions that I was used to making pretty much only impacted my deliverables (the users of my products didn’t care about the internals, just the results). It helped I was mostly a sole-contributor of code and not part of a larger programming team so I could refactor as I learned about the solution.

But now? Umm. I was part of a team that was responsible for the security of a large company; one of the oldest, possibly the largest, fintech company in the US; one that handled almost half of all credit card transactions in the US; one that ran the largest debit card network in the US. If I advised teams wrong and we got compromised as a result… I didn’t want to think about the consequences.

As time went on I grew a bit more confident. I also encouraged the rest of the team to chat a lot on IM (we were distributed across the globe; I was the only one in my location). This chat group was for both random nonsense, griping at management, laughing at bad vendor presentations, and also a “help” resource; if I wasn’t sure about something then I could ask the team for their opinion.

But I didn’t have anything concrete I could point to; “I did this”.

Worse, as I got exposure to more parts of the company, I got the feeling I was annoying more and more people. I tried not to be a “no, you can’t do this” type person and tried to help them come to a good solution, but there are always problem teams.

I’ve been a blocker, more than once; I’ve demonstrated solutions had problems (one team leading to the CISO having words with a VP of another team); I’ve probably antagonized people. I know I’ve spent many a sleepless night worrying about meetings I’ve had, or were about to have, and how they could go better.

Self deception, self deprecation

Because of a lack of hard deliverables, I had no way of measuring myself.

Of course my year-end self evaluation was full of “I did this”, “I did that”, “I did the other” but these were just words.

I find it hard to accept praise. My boss said I was doing a good job. More words. Apparently the CISO was happy with my work (although I rarely spoke directly with him). Yet more words. Where was the objective truth?

So I would joke, downplay my contributions, say things like “I have opinions on lots of things; for some reason people want to listen to them”.

I really didn’t believe I was making things better; at best I was just stopping things getting worse.

Yet I get tributes

After I announced I was leaving the company and taking early retirement a number of people sent me messages (both in public and in private).

you have always been a voice of reason
I learned more from you than you realize. I truly appreciate everything you’ve done.
you and your brilliance will be greatly missed.
definitely learned a lot by just listening
will miss your expertise and guidance
I’ll miss having you around Stephen, always a pillar of reason and knowledge
I cannot think of a time when I disagreed with you at first, that by the end of the conversation you had not convinced me with facts and logic to move my position. THANK YOU FOR THAT! So few people here do that
I also want to thank you for the times I was able to work with you. At least for myself, as you had hoped, I did learn a lot. Not in as much around “facts” or raw technical knowledge (there was plenty of that) but how to look at a problem, problem solve and think it through to a solution (or as close as we could get to one
I will personally miss you and even though we don’t collaborate as much as I want lately (my fault, not yours), I somehow feel that part of “the good” of the company will now be missing.
i will miss the expertise and that you don’t succumb to a flashy dashboard and ask the hard questions
You will be missed - you are one of the wiser / smarter people here.
It’s sad that we couldn’t work more together. Really enjoyed that and it’s definitely a loss for the company.
So many thoughts, sadness for losing you as part of this team, gratitude for everything we’ve built here together, excited for you getting out of this place and hopefully enjoying a well-deserved retirement. I can’t imagine where I’d be without you, and am humbled everyday seeing how we’ve both learned and grown together the last (almost!) 10 years.
The legendary journey has been etched in history. It was a privilege to cross path with you.
Ha. I don’t mind you bugging me. You are at least well reasoned and thorough

There were also some tributes paid on the weekly leadership call, from SVPs and VPs. I didn’t record them verbatim, but they included things like

Helped me immensely
We wouldn’t be where we are without Stephen. He changed the way the cyber org works, and we created a whole new team because of him.

Summary

It wasn’t until I was leaving that I started to believe what people were saying; people I thought I had annoyed, people I had said “no” to, people who I thought just ignored me or saw my role as a speed bump stopping them working… all of them were giving me unsolicited praise.

Which really highlights the difference between my self evaluation and the reality. Right up until the end I wasn’t sure I was doing a good job, nor making things better.

In the long long distant past of usenet I had a signature:

      The truth is the truth, and opinion just opinion.  But what is what?
       My employer pays to ignore my opinions; you get to do it for free.

Maybe I was wrong and people did pay attention to my opinions after all!

I still dunno what I want to do

Sun, 07 Apr 2024 19:14:25 -0400

Recently I wrote about how I got here without knowing what it was I wanted to do.

That was a prelude to the other half of the equation; I may not know what I want, but I do know what I don’t want.

At this moment in my life, I don’t to work. At all. I want to have the luxury to be able to lie in, to read a book, to stay up late hacking on some code or whatever…

I find myself resenting having to get up to work. And I had a fantasy of deleting work apps off my phone and how happy that made me.

So I quit. I’ve not just quit this job, I’ve quit the whole concept of work. I’ve taken early retirement.

So how can I afford to do this, at the age of 55?

I got lucky

Obviously a young white boy in England, even from a working class background, in the 1980s had benefits, including a world class free education; I got to go to Oxford! The first of my family to ever get to University. And that led to sponsorship and paid summer work, and my first job and…

Being smart helped, of course. Not falling into the wrong crowd. Having parents who both worked and were able to support me during my education and were happy with me being at home. Selling my UK house and transferring the money to the US at a peak pound/dollar rate meaning I was able to pay off my US mortgage.

Yeah, I got lucky.

Frugal. And FIRE.

In other respects I’ve been planning this for a couple of decades or more. I’m not sure when I first thought about retiring at 55; moving to America definitely put those plans on hold ‘cos I didn’t know if this was going to be permanent or temporary (fortunately the relationship with my darling Lady is still going strong, so it became permanent!).

But I’ve always spent less than I earned. I lived with my parents until I was 30, so was able to save up a good percentage of the purchase price of my house (smaller mortgage, easier to pay off sooner). When my salary increased above inflation (promotion, new job, whatever) my outgoings didn’t really go up to match.

Heh, I even remember as a kid when we went on holiday to the Isle of Wight, I’d saved up my pocket money so I had 20p per night to spend on sweets and stuff… and I would buy a 5p bag of gobstoppers. At the end of the two weeks I had enough money to buy stuff at the camp gift store.

So I haven’t bought fancy cars, I don’t buy expensive whiskey, I count every penny, almost literally! I wrote programs to track all my expenditure and could tell you, to the penny, where everything went. I still use those programs today! I would agonize over every “luxury” expenditure (where luxury might have been $50 or less).

Today, at age 55, I earn really good money. But my total outgoings were the third lowest in any year in over a decade. One of my programs shows yearly outgoings by category over the past 12 years; last year was a lot less than 2011, despite things going up (eg property taxes, house insurance).

It definitely helps to have paid off the mortgage and not to have a wife or kids; those three things can really increase costs. As a result, I’ve been saving a lot.

Apparently this is now known as FIRE (Financial Independence, Retire Early). I didn’t quite do Lean FIRE, but not too far short!

I’ve written more programs and tried to have been conservative in my estimates (higher than average inflation, lower levels of investment returns, higher amounts of tax… and high levels of medical expenses). I estimate my savings should last at least 45 years, possibly 55 years or more. And that doesn’t take into account what I’ll get on that sad day when my Mum passes on.

The future

I now feel confident enough that I can afford to live without any new income, and that resentment I feel each morning doesn’t have to be put up with. So why suffer it?

As for what I’m going to do… “I dunno”. Story of my life.

I’m not a big one for travel. I’m also not a sun lover (indeed me and the Sun don’t get along; it gave me skin cancer!). So I don’t expect to do much of that.

I have a backlog of techie things I’d like to do; incomplete projects, unwritten code, stuff like that. Both retro computing (my BBC Micro) and modern.

Maybe I’ll actually do some NYC tourism; I’ve lived here for over 20 years and not really done the tourist thing!

Or maybe I’ll just laze around and re-read the 1400+ SciFi books I have on my shelves, re-watch the 3500+ DVDs (movies and TV shows), learn how to play video games (beyond Backgammon).

And maybe… maybe I’ll get bored. If I do then perhaps I’ll dip my toe back into the workforce; consulting, perhaps?

Last day at work

I’ve given a couple of months notice; I expect my last day to be May 31st.

The first thing I’ll do, after that, is turn off the alarm and have a lie in.

New Trek

Thu, 04 Apr 2024 20:43:17 -0400

So I don’t do much “streaming”. I’m old school; if I like a show then I’ll buy it (or get it as a present) on physical media… BluRay or 4K these days.

This meant I hadn’t seen much of New Trek. A few years back I got Picard Season 1 on BluRay, then season 2. But this year I’ve received/bought a lot of the new stuff, and I thought I might write down my thoughts and opinions. I’m going to do this in (mostly) the order I watched them in.

Note that everything is my opinion. I’m not the God of Trek; I’m not even part of fandom. I’m just a fan (if you don’t grok the difference, don’t worry).

Picard

Season 1 was not Star Trek. It was generic SciFi with a cut’n’paste Trek overlay. It wasn’t bad SciFi, it just wasn’t Trek. It could have been so much more.

Season 2 was worse. Oh let’s explore Picard’s past and hidden trauma. WTF? Where did any of this come from. This was terrible.

Season 3… ah, now we’re talking. Yes, it leaned very heavily on TNG to hit its emotional points. But it worked. The flashback scenes worked well, the character interactions worked… it worked. The attack vector… OK, Trek has never been the most consistent; I’ll give that a pass. Not the worst technobabble the show has ever produced! And a lovely callback to TOS. You can (mostly) watch Season 3 without 1 and 2, and enjoy it.

Actually, yeah; skip seasons 1 and 2; watch 3.

Short Treks

I got this on a whim. And I’m glad I did. These are short stories (it’s in the name) but they’re very very well done. Some of them are preludes to later series, but they’re not necessary pre-watching.

There’s some good fun stuff here. And some stories with a big emotional impact. Great story telling. Recommended.

Prodigy

So I caught an episode of this on Nickelodeon and thought “wait, what?” and programmed my TiVo to catch all of them.

This is a kids animated show. But it’s Trek. It’s GOOD Trek. So 11 episodes.. where is episode 12? (time passes). C’mon, where’s the rest? (google google) Oh, it’s only on Paramount+. Oh, and they deleted it, so you can’t even see it there? Oh yay, it’s on BluRay… Christmas present!

And, yes, the second half of the season is just as good as the first.

Why the hell Paramount decided to cancel (and delete) the show boggles my mind; this was excellent story telling. Grump.

Strange New Worlds

I watched the first episode and realised it came after Discovery so I quickly ordered that and put this on the back-burner…

Discovery

I wasn’t sure what to think of this going in. I’d just had the call-back from SNW. OK, it’s pre-TOS which is always something to be careful of (because of breaking the very fragile continuity there is). And it kinda worked; yeah yeah the control panels are far advanced of what TOS had, but that’s just a “visualization”; it does the same thing, it works the same, it’s just a touch panel rather than physical crap.

And the magic drive system… yeah, how they gonna not break continuity there? Let’s leave that aside…

Season 1 is good Trek. It fills in a part of the time line that doesn’t really contradict anything. It’s well done. And they even have a rationale for why this magic drive doesn’t show up else where. Heh.

Season 2 is even better. This is a great show and a great story line. It starts with a puzzle and we follow the clues and encounter contradictions and problems and you think you know what’s going on (I’d guessed who the Red Angel was) we get a reversal that kills that guess… this is a great season. And the call-back to “The Cage” from TOS is excellent.

And then Season 3… it’s the consequence of season 2… and I didn’t feel it. I can see what they’re trying to do; I can understand the story. But it just doesn’t work for me. Far too much “post apocalyptic” type of story for me, and it wasn’t well told.

(I haven’t watched season 4 yet; that’s on my wish list)

Strange New Worlds (for real this time)

OK, now we’ve done Discovery the scenario makes more sense. Why Pike is who he is (his back story) is told in Discovery season 2.

But I’m afraid this isn’t a good show.

Well, it’s good Trek. It’s very good Trek. As a prelude to TOS (Pike was the captain before Kirk) it’s good. As an episodic TV show it’s good.

But that’s the problem; “episodic”. Every other New Trek (except Short Treks) has been a season-long story arc. Strange New Worlds is very much a throw back. It’s the same reason I consider voyager to be the worst of the original Trek era.

(To be totally open; I still have 4 episodes of Season 2 to go, but I’m not expecting things to change based on the previous episodes)

Every episode of SNW is excellent. But there’s no direction. When TOS was created episode TV was the norm; in the 2020s we expect more. And SNW doesn’t give that “more”.

SNW could have been a perfect bridge between Enterprise and TOS; Discovery had already set the pattern. But no real story arc? Disappointing.

Conclusion

I’m the sort of person who if they like a show then they’ll want it on physical media. If I did have Paramount+ then I’d still have bought most of this.

None of the New Trek is bad.

If I was to make a recommendation, I’d skip Picard seasons 1 and 2. Strange New Worlds is missable, but it’s no worse than a lot of shows. Prodigy and Short Treks are unexpected fun.

I haven’t seen Lower Decks (yet!) so I can’t comment on that.

I dunno what I want to do

Mon, 25 Mar 2024 13:39:36 -0400

One of the most annoying interview questions is “where do you see yourself in five years time?“. I hate it. I have no vision of the future like this. Hell, I barely know what I want to do tomorrow.

I’m good at foreseeing the future, honest!

So my first job, straight out of uni, was with a small Greek shipping company. I learned a lot there ‘cos I had to do it all. But I thought the internet would be a fad; you can do all you want with email and UUCP and store’n’forward systems. Why would we need complicated TCP connections to every desktop? That’s unnecessary.

My second job was at a web publishing company (oh, look; internet!), where I went from Unix SA and networking to Oracle DBA to Windows SA to Lotus Notes Admin… there’s no way I could have foreseen that path.

My third job quickly led to an approximately 50% salary increase in a matter of months (promotion). So that house I had just bought… I could pay off the mortgage in 10 years, rather than 25.

But then I fell in love, and moved to America. Yeah, like that was on my plan! 5 years? That wasn’t even within 2 years.

I stayed at that job for 17 years, but kept moving departments (SA, engineer, automation, config reporting, access management, security engineering, cloud security…) whenever I started to get bored, or whenever the organisation needed me to do something else. The advantage of a good mega-corp is that there are plenty of internal movement opportunities.

My fourth (and current) job came out of the blue as the result of a call from a friend asking me for help.

None of this has been planned. I just drifted around; solve this problem, solve that problem then move on (hopefully leaving the place better than when I started) when I start to get bored.

Doing stuff I enjoy

I remember telling my boss in my second job that I was thinking of leaving; he asked if there was anything that could be done; what would I like to do, what could they do to keep me. I had no idea, no answer. I told him that if I knew what it was that’d keep me interested then I’d already be doing it.

That’s not too far off. I’ve always been pretty lucky to be able to do skunkworks type stuff; play with things and then tell other people about it. Normally they found it useful ‘cos I would build it to make my life easier, so it solved problems that other people were also having.

Looking back to look forward

If I look back at my career then I can see a pretty good progression; although none of it was planned. Even my very first job was unplanned; my university tutor mentioned that a lecturer in another college wanted to talk to me; that lecturer had a friend who was looking for a Unix SA for his company; I interviewed and got the job.

All the different roles I’ve had mean that I get a better holistic view of problems; I’m not stuck in a single mindset rut, not focused on a single technology solution. I’ve seen a lot, made a number of mistakes, learned from them and can put all this together to design new solutions or spot gaps/problems with proposals from others.

And it’s by looking back that I come up with answers to that annoying interview question.

So in that second job I’d brought in Big Brother to monitor the web servers and network. Then I wrote some Kermit scripts to talk to the pager gateways so BB could page me if there was a problem. Which I then opened to the whole team so they could page each other without needing to phone the paging center. Then I added more monitoring, and added Netware pings (so the Netware team got alerts when their aging repurposed desktop-as-server machines fell over).

My first job I managed 3 machines; my second job 20 machines. When it came to my third job interview I was able to describe that automation and monitoring, and said I wanted to automate and manage bigger environments; I ended up starting by automating 550-ish servers. When I left that job my software was running in 65,000 servers.

Which then led to the answer for my fourth job; “I’ve done scale from tiny to mega; now I want more complicated problems”.

They were true answers, but they weren’t the whole truth. I didn’t really know what I wanted to do in 5 years; I just don’t do that type of planning.

Privilege

I try to take care that my decisions aren’t overly risky (although that move to America was definitely one!) so I don’t break my potential future, but that’s my normal conservative approach to everything.

But I’ve got to acknowledge an amount of privilege at work, here.

I’ve never failed, but I knew I had a safety net if I had (especially in the early days); I lived at home until I was 30 so never had to worry about keeping paying rent or getting food on the table; I lived in the UK so didn’t have to worry about expensive medical bills; I’m smarter than the average bear; and I got lucky.

Conclusion

It’s clear that I’m not a visionary, and that I’ve never had a plan. I’ve never said “this is where I want to be”. I’ve just gone where the tides have taken me.

So now I’m asking myself… what do I want to do tomorrow? And the answer is “I dunno; let’s see what comes up!”

And that’s OK.

For me.

MQTT, Home Assistant, Hue emulation working together

Mon, 11 Mar 2024 17:11:06 -0400

A number of years ago I wrote a Hue Bridge Emulator that would let you emulate light bulbs in shell script in such a way that these devices could be controlled by Alexa (and so used in routines and the like). It worked well.

But recently Amazon appear to be changing how hue bridges are detected. The big challenge appears to be it wants the server to listen on port 80. This is annoying for a number of reasons, including that it needs privileges to bind to that port, and it may conflict with a web server already on that port.

Now I also run Home Assistant which has an inbuilt Hue emulator. So I figured we can use that functionality, and let you control more things through HA which don’t normally have integrations. A simple example might be turning your screen on/off; have a “bed time” routine that turns your screen off!

We do this by configuring a light in a MQTT configuration, and have code to subscribe to topics so it sees when to turn/off or (optionally) change brightness. It will pass this data down to your program (which could be a shell script) and update MQTT status topics with any feedback from the program.

In theory this could all be done from shell scripts (eg using mqttcli commands and coprocesses and more) but I feel this approach is simpler; your program only needs to deal with stdin/stdout and doesn’t need to care about MQTT at all.

Configuring Home Assistant for MQTT

You need to set up (if you haven’t already) the MQTT integration. Running an MQTT broker and configuring the integration is beyond the scope of this doc. I, personally, use mosquitto as the broker; it comes with many Linux distributions and is easy to install.

Now we need to define a light. This is done in the mqtt: section of configuration.yaml.

After manually configuring MQTT instances the way I’m describing here you will need to tell HA to reload the configuration (e.g from Developer tools), or else restart HA totally.

There are two kinds of lights we can model:

On/Off only

mqtt:
  light:
  - name: "Test Light"
    command_topic: "mqttlight/test/command"
    state_topic: "mqttlight/test/state"

With this light it will create a new light. entity in HA (probably light.test_light). If you click the ON button it will send an “ON” message to mqttlight/test/command. The state of the button will be reported back to HA on mqttlight/test/state.

On/Off with brightness

This emulates a dimmer switch; it has on/off and brightness controls.

mqtt:
  light:
  - name: "Test Light"
    command_topic: "mqttlight/test/command"
    state_topic: "mqttlight/test/state"
    brightness_command_topic: "mqttlight/test/bright_command"
    brightness_state_topic: "mqttlight/test/bright_state"
    brightness_scale: 100

Exposing these lights to Alexa (maybe even Google?)

Once the MQTT entries have been configured you will be able to see the entity IDs in Settings / Devices / Entities. If you search for the name you set you’ll see it show in the entity list. We’ll need this.

Now in configuration.yaml we need to tell HA to expose these lights to Alexa, via the emulated_hue configuration. I normally don’t expose every device, just the ones I select:

emulated_hue:
  listen_port: 80
  expose_by_default: false
  entities: !include emulated_hue.yaml

This lets me use a new file emulated_hue.yaml to list the devices I want. You don’t need to have it as an included file, I just find it simpler this way.

The contents of this file are now the entities we want to expose.

light.test_light:
  name: "Test light with brightness"
  hidden: false

The entity ID is the one we found earlier; the name is the how we want this light to show in Alexa.

That’s it! That’s the whole emulated_hue configuration.

Changing this file requires a full HA restart.

The emulated_hue setup can be made a lot more complex; such setups are described in the documentation.

Once you’ve started it you should be able to confirm the bulb shows up by talking to the emulated hue endpoint.

e.g.

% curl http://your_ha_ip_address:80/api/v2/lights | jq .
{
  "1": {
    "state": {
      "on": true,
      "reachable": true,
      "mode": "homeautomation",
      "bri": 1
    },
    "name": "Test light with brightness",
    "uniqueid": "00:42:af:28:ad:f5:e0:ea-de",
    "manufacturername": "Home Assistant",
    "swversion": "123",
    "type": "Dimmable light",
    "modelid": "HASS123"
  }
}

(jq is a nice command that make the JSON look pretty).

You can now tell Alexa to discover new devices… and it should(!) work.

More complex emulated_hue setups are described in the HA documentation.

Communication between bridge and program

The communication patterns are asynchronous. The bridge can send commands to the program and the program can send status updates back to the bridge at any time. It’s a very simple protocol.

Commands:

There’s only one command:

LIGHT#name#on/off#brightness – Set the light named “name” to the on/off status and the brightness (0-100). The name must not contain a # since that is used as the separator. If either the on/off or brightness values are - or missing then that value should be left unchanged.

examples:

LIGHT#test##70
LIGHT#test#on#
LIGHT#test#on#70

Status Updates:

The child can send two messages to the bridge. These are best sent periodically (e.g. every 5 seconds).

LIST#name1#name2#name3 ... – This can be used to tell the bridge of the complete set of lights being controlled. For example, if you have an environment where devices may join and leave (a mobile phone, perhaps?) then this can be used as a way of refreshing the bridge’s knowledge and to stop telling clients about lights that no longer exist.

The bridge uses this list to filter out what messages seen on MQTT topics should be sent to the child program or not.

LIGHT#name#on/off#brightness – This is the current state of the light. When this is sent it will trigger an update of the relevant MQTT topics. The brightness may be left blank if it’s not relevant. If this is a new light then it will be added to the list seen from the LIST command.

e.g.

LIST#monitor
LIGHT#monitor#ON#

Importantly, the name of the light sent in the LIGHT command will be used to create the topic name. e.g. LIGHT#monitor#ON# will cause an ON message to be sent on topic mqttlight/monitor/status. This is case sensitive; “monitor” is different to “Monitor”

Running the program

Usage of ./mqttlight_bridge:
  -app string
        Application to run
  -base string
        MQTT Base (default "mqttlight")
  -debug
        Allow CLI entry to child
  -pass string
        MQTT Password
  -port int
        MQTT Port (default 1883)
  -server string
        MQTT server (default "localhost")
  -user string
        MQTT Username

The -app value is mandatory, the rest are optional. If your MQTT broker requires a username and password then these should be specified.

Examples

Two simple example scripts are provided in the source repo.

small-example - acts as a simple light called “test” which can be turned on or off and have the brightness changed
screen_on_off - an On/Off light that uses xset q to determine if the monitor is on or off, and xset dpms to change the state.

Possible gotchas

I’ve sometimes seen that changing the brightness from the Alexa app may cause two updates to be sent; the brightness value and then an ON message. This would cause two commands to be sent to the child

  LIGHT#name##brightness
  LIGHT#name#ON#

you should be careful that actions taken in response to “ON” without a brightness value should not change the brightness by mistake.

I’ve seen Home Assistant get confused; I originally created an mqtt light as on/off and then later changed it to one with brightness. HA then wasn’t sure how to present it; the GUI presented on/off and brightness slider, and the slider did change cause a brightness command to be sent… but it never reflected the actual brightness. I found I had to stop HA and edit the .storage/core.restore_state to remove the existing entry for the light before it would pick up the new definition.

Building the code.

This uses pretty simple GoLang and so should compile and work on any of the main supported platforms (Linux, MacOS, Windows, etc). You obviously need the GoLang compiler and git.

An example session might look like:

$ git clone https://github.com/sweharris/mqttlight_bridge
Cloning into 'mqttlight_bridge'...
remote: Enumerating objects: 18, done.
remote: Counting objects: 100% (18/18), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 18 (delta 2), reused 15 (delta 2), pack-reused 0
Unpacking objects: 100% (18/18), done.
$ cd mqttlight_bridge/
$ go mod tidy
$ go build

And that’s it! The go mod tidy may download modules from github and golang sites if the dependencies are not present.

You can now use the program; e.g.

./mqttlight_bridge -app ./screen_on_off -server mqtt_server

Conclusion

I have a love/hate relationship with Home Assistant. I think it’s a very useful program and my home automation is totally dependent on it. I’m just that much of a fan of its internal architecture; I would much preferred a decoupled setup. Indeed, using MQTT in this way is very decoupled; we can restart the bridge code and the child app and Home Assistant doesn’t care.

I’m annoyed that Amazon keeps changing what they expected for a “fake hue” setup, but at least Home Assistant can fill in the gap!

Blame Security

Sat, 05 Aug 2023 18:04:04 -0400

Else-net there was a discussion on how “security” is generally seen as a blocker; they’re seen as gate keepers and people who just say “no”, or who may be focused on regulatory compliance and not actual security.

Who needs Mordac, the Preventer Of Information Services when you have a security team?!

The thing is, “security” isn’t a monolith, and it’s not a one way street.

Many teams

I’ve only really seen security from a megacorp perspective, both from the companies I’ve worked for and the people from other companies I’ve spoken with over the years.

And there is no one security team in these situations; there’s a whole diverse group of teams; maybe all under the CISO, maybe not!

So there might be

a team of risk managers (look at gaps, evaluating them, documenting risks, track remediation).
a team for firewall governance (ensure there’s no free-for-all access in/out of the company)
a team handling application security (enabling shift-left, if they’re good!)
… handling email security
… for encryption
… for network security (honeypots, deception, packet flows, etc etc)
… WAFs
… Identity and Access Management
… data loss protection
… data discovery
… SOC
… Incident response

… and more.

Each company will have its own structure, of course. It’s not fixed in stone.

Regulatory compliance is really a small part of all of this, and may not even be under the CISO. In my opinion, a company focused on compliance isn’t a company that cares about security. They will be breached because they see it as a box ticking exercise.

Now some of those teams may run their own technology, others may use tools provided by the core technology area (e.g. firewall governance team may merely control the rulesets with the technology network team actually running the firewalls).

I work in security architecture. I’m a generalist. Part of that job is to ensure these security teams are doing things that further the security goals of the company and that all those teams aren’t duplicating efforts (‘cos strict delineation of areas isn’t really possible; there’s always overlap; e.g. email security team has a DLP requirement for outgoing email), and to provide consulting services (e.g. to the risk managers to help them understand the problems so they can provide better evaluations).

And, of course, in a megacorp for some teams are better than others.

Which leads to another part of my job; working with the business units (BUs) to assist them in building suitable solutions.

Over the years I’ve found there are two main types of BU teams; those that want to do the right thing, and those that just want to deploy as fast as possible and just see security as a hindrance.

The first kind I work well with; I can explain my logic, they can explain their solutions, we can work out what needs to be done to be secure (rather than just compliant). I rarely need to point at the standards and rules with these teams because they care. And we build good solutions together. We have a virtuous circle; they’re more willing to come to me early in their design cycle and we get better results, faster. A recent comment from BU team member was long the lines of “Thanks for the long and detailed response; that makes perfect sense, and I think I can see a solution path for our use case”. I’ve had an SVP from one of those teams call me the best security person he’s ever worked with. But this only works because they are willing to take that extra step.

The other type of team, though… they’re a problem. If it’s not in the standards they won’t do it. They push back because it might be hard, because it’ll cost money, because they just want to meet deadlines, because they’ve never been held to account before and they have over a decade of “do their own thing”. These teams I sometimes have had to rule by dictate. If they’re not willing to work with me to help define a better outcome then I have to be the blocker. This is a vicious circle, and one I dislike.

In some ways we can classify these teams as those following the spirit of the law (“we need to protect our data and systems”) vs those that follow the letter of the law (“we need to be compliant”). I so much prefer the first.

At the end of the day my goal is to help the BUs operate in a secure manner; I’m not here to block, I’m here to help. But I’m not able to force them to engage constructively; “tell me where in the standards I have to do this” is not conducive to a meaningful discussion. In the past I’ve joked that my job is saying “Don’t be stupid; don’t be stupid; I said don’t … oh God, you were stupid weren’t you?“.

Every mega corp will have both types of team. Fortunately the second type is getting less prevalent, as technology changes. For example, teams that are refactoring for cloud or for k8s or for automation tend to want to use modern CI/CD processes and want to do it correctly from the start.

Hide behind security

Sometimes teams blame security even when it’s nothing to do with us; it might be an easy way of punting work that’s hard, for example.

Or teams may interpret requirements in the worst way and assume there’s nothing they can do; “We can’t do FOO because the standard says BAR”. That’s even worse when that team provides services to other teams (e.g. central technology teams providing services to BUs) because the impact is magnified.

Security can also be the front man for other teams; rules and requirements can come from legal and compliance teams (who aren’t security). We get the blame for stuff because we’re the ones who have to implement it; “No you can’t record Zoom meetings because security said turn it off”; it was legal who said turn it off ‘cos Zoom recordings didn’t meet their records retention policies, we just had to implement it for them.

Security can end up getting the blame for things that aren’t our fault.

Summary

I tell people I don’t really care too much about compliance; good security practices (mostly) result in compliant solutions anyway! So I try to push for good security practices. Okay, I do keep some primary stuff in mind (e.g. PCI) but most of that is baseline bare minimum; you can be PCI compliant but not secure.

But I can’t do it alone.

Security can only collaborate with teams that want to collaborate. I need teams to work with me; a “help me help you” type scenario.

Maybe it’s not all my fault, after all!

Digital Safe version 3

Tue, 13 Jun 2023 18:01:27 -0400

Previously I had modified a digital safe to be controlled via an ESP8266; basically a WiFi safe.

I was asked if I could create a firmware for it that made it act like a timer safe; something along the lines of a Kitchen Safe.

I decided to take the opportunity to build (yet another) safe, using the combined esp/relay board. Without any soldering, I’m sure I can make a cleaner more reliable build!

Requirements

A cheap Safe (Amazon link)
An ESP8266 relay board.
A 1N4001 Diode
Cables
Power Supply

The main relay board looks something like this

I’m not going to link to a seller, ‘cos sellers seem to change a lot. I’ve found searching “esp8266 hw-622” tends to find sellers.

This board comes in a couple of versions; the biggest difference is whether headers at the top left are present. I recommend getting them pre-installed because these boards don’t have USB connectivity; you program them using a 3.3V serial adapter connected to them.

The Safe

This time I picked a safe with a slot in the top. The normal idea is that the safe can be locked and then money added. It’s about $33. There’s a lot of similar models on Amazon.

The internals on this safe are a little different to what I’ve seen before. Previously there was a simple plunger solenoid. This time the lock rotates.

In the locked position it looks like this.

When almost totally open we can see it’s rotated down out of the way

This would appear to fix a major issue of the previous design; the old one is susceptible to hitting it hard to send the plunger down, or even a strong magnet.

Of course the metal is still very thin so it’s not a high security safe!

Open the safe cover and disconnect the solenoid wire as we did in the v2 safe. I made a small hole in the door near the battery compartment so it can exit there. That’s it. That’s the only modification!

Put the cover back on.

At this stage I recommend programming the esp8266 with the safe software. It’s easier to get to the contacts.

It works just fine with the existing software

To do that we need to power it. So let’s talk about this. These boards can typically be powered from 7V up to 20V, if not higher. The power supply will also power the solenoid. I measured the coil at 18 ohms, so I think any power supply 500mA or higher would do; maybe call it 750mA for safety?

Now I had some spare 2.1mm x 5mm power pigtails spare.

These aren’t optimal for this job because they’re not designed for “panel” work, but they do the job.

Connect the red wire to the right side, and the black to the left

Now you can program it using the Arduino software and your 3.3v serial adapter.

Hook it up to your network and test the safe software works. You’ll hear the relay clicking. Remember to change the configuration so it uses “4” (GPIO4) as the trigger.

Putting it all together

First I jammed the power cable through one of the mounting holes on the back. This was harder than it should have been because it’s not a panel mount option. And the hole was just a little too small so I stuck a screw driver in and bent the metal slightly to make it bigger.

It still sticks out but it’s good enough.

Now we need to add a diode to the output. This is to prevent “back current” from potentially damaging the ESP8266 (when I tried without the diode the board would reboot maybe 1 time out of 3). Almost any diode should work; I used a 1N4001. This needs to connect between the convenient 5V output at the bottom of the “common” on the relay. The stripe side of the diode should be at the relay end.

Now we just need wires going from the negative power at the top of the board and from the far right relay connection at the bottom of the board to go to the solenoid cable.

And that’s really it!

This is what a diagram of the wiring looks like. Did I just scribble something on a piece of paper? Yes, I did!

Yeah, there’s two wires going to the negative power; just jam them into the same hole.

To mount this I used some 3mm plastic stick-on standoffs. And I figured I could stick it to the roof of the safe!

This makes the whole thing very clean.

This is a lot simpler than the v2 build; no soldering, no holding it together with duct tape; just a few wires!

Software

As mentioned, the old software will work; just set it so GPIO 4 is used for the relay.

But we want a timer safe.

The problem I had, here, was in determining the current time. Of course we could just use NTP, but if I wanted to defeat the lock then I’d just set up DHCP to point to a “rogue” time server that claimed it was in the future, and the safe would open. That’s not good.

I realized I didn’t need to save the time, just how long the lock had to run for; time conversions could be done in the browser. Of course we still had to worry about power outages, so every 5 minutes I would save the “time left” into the EEPROM.

Except… Every 5 minutes would be 288 times a day. ESP8266 doesn’t have EEPROM; it emulates it with a fixed space in the flash memory.

I saw that the flash typically has 100,000 write cycles. That means we’re reaching the limit in 347 days; less than a year.

It seems that the SPIFFS (and LittleFS) filesystem that can be used with the ESP8266 has built in wear leveling. Using the 4M1M memory model this allocates 1M to the filesystem. Assuming we use 512 bytes to store data and the meta-data, good wear leveling should means this lasts over 1000 years (if I did the maths right!); at the very least it should exceed the life of the safe.

So that’s what I did; saved the “time left” value in a SPIFFS file.

I then thought I could be clever; why not provide more “fun”… instead of setting an “open time”, why not allow for a time range and pick a value between those times? And for additional uncertainty, why not hide the timer?

Of course time can be added to the safe even when it’s running.

This software can be built with the standard Arduino tool set with the esp8266 libraries loaded. To make it simpler I embed the HTML inside the code (as an include file) rather than using SPIFFS. That way there’s just one image to flash and not two. Yes, we’re using SPIFFS to store time left, but in this way we can update the software easily (even over the network) because it’s just the one image. The code is small enough that it still fits.

The solenoid lock is always kept in the locked position unless the “open” function is selected. Open can not be selected if the timer is still running.

When the safe software is first loaded it will create a WiFi Access Point called something like “Safe-123456” (the exact name will depend on the MAC address of your board). You should connect your phone or laptop to that access point.

This will let you go to http://192.168.4.1 or (if you’re lucky) http://safe.local and that will display the WiFi configuration screen.

Enter your WiFi SSID and password. In 5 seconds the software will reboot and the safe will connect to your local network. It will attempt to get an address by DHCP.

If it is unable to connect to the network after 60 seconds then it will give up and start up the Safe-123456 access point again and allow you to change the network details and try again.

Once the safe is on your network it will broadcast the name “safe.local” via mDNS (aka zeroconf, Bonjour, Rendezvous). If your machine supports this then you should be able to go to http://safe.local otherwise you may need to check your router to find the IP address that was assigned to it.

Opening the web page in a browser should give you a menu on the left and a larger status box on the right.

The first thing you should do is go to the “Change Safe Authentication Details” link. Here you can set a username and password for your safe. It’s recommended you do this so that only authorised people on your network can reach this interface. After setting the password you should be prompted to login before doing any more work.

Next you may need to go to the “Safe Connection Details” screen.

Here is where you can change the WiFi network (if necessary) and the pin the relay is connected to. For the board I’m using in this build this needs to be set to 4 (GPIO4 aka D2). The v2 board used GPIO12 (D6).

Setting the “Safe name” will change the name broadcast via mDNS. So in this example my safe can now be seen at “safe3.local”.

Elsewhere on the menu you’ll see some options:

Open safe” lets you open the solenoid so the door can be opened. This will not work if the safe is locked.
“Enable Safe Network Update” will let you update the firmware remotely, using BasicOTA. If you set a safe password (as recommended above) then the same password is needed for this update. Now you can’t do an update if the safe is locked because then you could just replace the software with one that just always opens the safe!
“Disable Safe Network Update” turns the enable off again

And then the main “Status” screen. This is where the action occurs.

Here you can add time to the safe; if it’s already running a timer then this time will be used to extend the lock. If you just enter a “Min” set of values then this is the time that would be used. If you enter a “Max” then the safe will pick a random time between those two values.

While locked it will tell you when the expected unlock time is, and how long is left.

But if you select the “hidden” option then you won’t know…

Conclusion

And that’s it!

The new software is at github

It seems to do what the original Kitchen Safe does, plus more. And possibly cheaper!

Terrahawks Noughts and Crosses

Sat, 27 May 2023 21:14:49 -0400

Yes, this is a blog about a very old TV show.

I went down a rabbit hole. A very stupid rabbit hole. A meaningless rabbit hole.

There was a 1983 Gerry Anderson puppet show “Terrahawks”.

It wasn’t as good as his older stuff (e.g. the original Thunderbirds). Maybe I was also that little bit older when I first watched it (I would have been around 15 when it started).

Earth was under attack by an alien, Zelda. Earth’s defenders has robotic spheres (“Zeroids”). Zelda had cubes.

At the end of each episode there was a game of noughts and crosses, with the zeroids playing the “O” and the cubes playing the “X” (‘cos they had an X mark on top of them).

Since I recently got the box set (it was cheap) I had to analyse the games. Joshua (Wargames) would wipe the floor with both teams, but let’s see.

There were 39 episodes. Each game started with the Zeroids (O) going first.

This make it easy to create a notation for recording the games; just number the squares and record the order they’re filled in. The first square is always an “o” and then they alternate “x” and “o”.

So the number 3975142 would result in this game. Yay zeroids win!

		o

		o

		x

		o

o		x

		o
	x
o		x

o		o
	x
o		x

o		o
x	x
o		x

o	o	o
x	x
o		x

Now to complicate things, the cubes cheat. they might move at the end and knock the zeroids out of place to claim a win. eg “2158647 8-7” Indeed, the only way the cubes won was by cheating like this.

	o

x	o

x	o
	o

x	o
	o
	x

x	o
	o	o
	x

x	o
x	o	o
	x

x	o
x	o	o
o	x

x	o
x	o	o
x

You can see an example video of this in action; this is from the first episode.

What I learned

It turns out there were only 13 different games. Episodes would repeat. I guess the animation was too expensive for 39 games.

Except there’s only 38 games, ‘cos one episode (“A Christmas Miracle”) didn’t end in a game, but with Sergeant Major Zero on his perch dressed as Santa.

Not every game was repeated evenly, either. The game in the first episode never got repeated; other games maybe 3 or 4 times.

All in all the score was 19:19.

As a kid I kinda hoped the winner represented who came out on top in the episode, but there really doesn’t seem to be any correlation between the game winner and the episode.

Which makes sense, since the Terrahawks were normally on the defensive.

What I missed

Now I had vaguely wondered about symmetry. Noughts and Crosses boards have rotational and reflectional symmetry. What I’d missed was that the character animations only had reflectional symmetry in the Y axis. So I could have sat down and worked this out, rather than being lazy.

When I posted my early learnings to twitter I was lucky enough for the actual artist to pick up on this and reply. How cool is that!

He confirmed that there were only 7 games animated; 6 were reflections and the first episode was the only one with a unique game. He even has a youtube video on this.

The games

Oh, if people care, these are the episodes for each game.

1459238

1.05 The Ugliest Monster Of All
2.06 My Kingdom For A ZEAF
3.04 Space Cyclops

o

o
x

o
x	o

o
x	o
		x

o	o
x	o
		x

o	o	x
x	o
		x

o	o	x
x	o
	o	x

1523746 6-9 3-6

1.01 Expect The Unexpected

o

o
	x

o	o
	x

o	o	x
	x

o	o	x
	x
o

o	o	x
x	x
o

o	o	x
x	x	o
o

o	o
x	x	x
o		o

1795362

1.09 Thunder Path
2.04 The Ultimate Menace
2.11 Cry UFO

o

o

x

o

x		o

o
	x
x		o

o		o
	x
x		o

o		o
	x	x
x		o

o	o	o
	x	x
x		o

2158647 8-7

2.08 Cold Finger
2.13 Ma’s Monsters
3.12 The Sporilla

	o

x	o

x	o
	o

x	o
	o
	x

x	o
	o	o
	x

x	o
x	o	o
	x

x	o
x	o	o
o	x

x	o
x	o	o
x

2358469 8-9

1.11 Mind Monster
3.06 Child’s Play
3.13 Gold

	o

	o	x

	o	x
	o

	o	x
	o
	x

	o	x
o	o
	x

	o	x
o	o	x
	x

	o	x
o	o	x
	x	o

	o	x
o	o	x
		x

3657218

1.03 Thunder-Roar
2.01 Operation SAS
3.07 Jolly Roger One

		o

		o
		x

		o
	o	x

		o
	o	x
x

	o	o
	o	x
x

x	o	o
	o	x
x

x	o	o
	o	x
x	o

3975142

1.10 From Here To Infinity
3.02 First Strike

		o

		o

		x

		o

o		x

		o
	x
o		x

o		o
	x
o		x

o		o
x	x
o		x

o	o	o
x	x
o		x

4517328 2-3

1.04 Happy Madeday
2.09 Unseen Menace
3.08 Runaway


o


o	x

o
o	x

o
o	x
x

o		o
o	x
x

o	x	o
o	x
x

o	x	o
o	x
x	o

o		x
o	x
x	o

5873149

2.02 Ten Top Pop
3.01 Two For The Price Of One
3.09 Space Samurai
3.11 Operation Zero


	o


	o
	x


	o
o	x

		x
	o
o	x

o		x
	o
o	x

o		x
x	o
o	x

o		x
x	o
o	x	o

5891367

1.02 Expect The Unexpected Part 2
1.13 To Catch A Tiger
2.05 Midnight Blue
3.05 Doppelganger


	o


	o
	x


	o
	x	o

x
	o
	x	o

x		o
	o
	x	o

x		o
	o	x
	x	o

x		o
	o	x
o	x	o

6539128 2-1

1.07 The Gun
2.03 Play It Again SRAM
3.10 Timewarp


		o


	x	o

		o
	x	o

		o
	x	o
		x

o		o
	x	o
		x

o	x	o
	x	o
		x

o	x	o
	x	o
	o	x

x		o
	x	o
	o	x

7453289 2-1 3-2 5-6 4-5

1.08 Gunfight At Oakys Corral
2.07 Zero’s Finest Hour
2.10 Space Giant



o


x
o


x	o
o

		x
x	o
o

	o	x
x	o
o

	o	x
x	o
o	x

	o	x
x	o
o	x	o

o	x
	x	o
o	x	o

9651287 2-3 1-2 5-4 6-5

1.06 Close Call
2.12 The Midas Touch
3.03 Terratomb



		o


		x
		o


	o	x
		o

x
	o	x
		o

x	o
	o	x
		o

x	o
	o	x
	x	o

x	o
	o	x
o	x	o

	x	o
o	x
o	x	o

No game! Picture of Zero dressed as Santa!

1.12 A Christmas Miracle

Summary

You can see each game is 7 moves long, and if no one (well, the zeroids) has won after 7 moves then the cubes cheat.

It’s also very clear to see the symmetry in the last 2 (“7453289 2-1 3-2 5-6 4-5” and “9651287 2-3 1-2 5-4 6-5”).

The show wasn’t as good as I remembered, but the music and how it sync’d to the end-game was clever and fun.

Apparently the American broadcast never got this. How unfortunate…

And now I have the end credit theme tune stuck in my head.

Oh, and if you want the source that generates the really bad HTML that draws the game boards, it’s here.

Using network namespaces for ipv4 only

Sat, 29 Apr 2023 14:23:35 -0400

The problem

I use the Ookla Speedtest CLI in a cron job to get an idea of the speed of my internet connection (Verizon FIOS), and spot if there are problems. Why? Because why not :-)

It let’s me draw graphs like this.

However, recently I was starting to get error messages that the command wasn’t able to reach speedtest.net to get the configuration. It wasn’t happening every time; sometimes it would go hours without issue, other times it would fail 3 or 4 times in succession.

When I looked into this, it turned out that the code was preferring IPv6 over IPv4. Now my home network is dual stack with IPv6 being provided by a HEnet tunnel. Because tunnels aren’t necessarily as performant as native networking I configure my machines to prefer IPv4; e.g. setting gai.conf

precedence ::ffff:0:0/96  100
scopev4 ::ffff:0.0.0.0/96       14

However it seems this program is ignoring that. When I ran speedtest manually I could see using the tunnel!

$ speedtest

   Speedtest by Ookla

      Server: Greenlight Networks - Binghamton, NY (id: 3156)
         ISP: Hurricane Electric IPv6 Tunnel Broker

Solution attempt 1

So my first attempt to solve this was to use LD_PRELOAD to intercept the name resolver calls and force it to only return IPv4 addresses. This is what gai.conf is meant to do (getaddrinfo(3)) but let’s force it.

Fortunately someone had already done the hard work so I just took their code. And it worked well with telnet.

e.g. LD_PRELOAD=$PWD/force_ipv6.so telnet speedtest.net 80 would force IPv6 and LD_PRELOAD=$PWD/force_ipv4.so telnet speedtest.net 80 would force IPv4.

Except this was wasted time; the speedtest binary is statically compiled (was it written in Go?) and so LD_PRELOAD has no affect. Oops!

Solution attempt 2

So if we can’t make the application work how we want, maybe we can change the run time environment.

This is exactly what Linux namespaces do. Namespaces are one the core underlying technologies that allow solutions like docker to run; it helps isolate applications from each other, and there are multiple different namespaces (cgroups, IPC, network, mount, PID, user, UTS).

For what we need here, a network namespace is all we need.

Network namespaces can be created and entered with the ip netns command. I’m going to call this namespace ip4only; it’s nicely descriptive!

ip netns del ip4only
ip netns add ip4only

In order to let the namespace talk to the rest of the network it needs a veth endpoint inside the namespace. This basically creates a virtual “point to point” connection, and you can place one end inside the namespace and leave the other in the “root” namespace. Let’s call the two network points ip4only-root and ip4only-ns.

ip link add ip4only-root type veth peer name ip4only-ns
ip link set ip4only-ns netns ip4only

Now each interface needs an IP address. Let’s use the 192.168.200.0/24 network. And while we’re at it we can add a loopback and a default route inside the container.

ip addr add 192.168.200.1/24 dev ip4only-root
ip link set ip4only-root up

ip netns exec ip4only ip addr add 192.168.200.2/24 dev ip4only-ns
ip netns exec ip4only ip link set ip4only-ns up
ip netns exec ip4only ip link set lo up
ip netns exec ip4only ip route add default via 192.168.200.1

So how does this look?

$  ip addr show dev ip4only-root
22: ip4only-root@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ee:f6:02:dc:35:90 brd ff:ff:ff:ff:ff:ff link-netns ip4only
    inet 192.168.200.1/24 scope global ip4only-root
       valid_lft forever preferred_lft forever

$ sudo ip netns exec ip4only sh
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
21: ip4only-ns@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 86:58:d8:c4:b7:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.200.2/24 scope global ip4only-ns
       valid_lft forever preferred_lft forever
    inet6 fe80::8458:d8ff:fec4:b747/64 scope link 
       valid_lft forever preferred_lft forever

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.200.1   0.0.0.0         UG    0      0        0 ip4only-ns
192.168.200.0   0.0.0.0         255.255.255.0   U     0      0        0 ip4only-ns

# ping -c 1 192.168.200.1
PING 192.168.200.1 (192.168.200.1) 56(84) bytes of data.
64 bytes from 192.168.200.1: icmp_seq=1 ttl=64 time=0.036 ms

--- 192.168.200.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.036/0.036/0.036/0.000 ms

# exit

It looks like it has an IPv6 address, but this is just a “link local” one and isn’t used for access to the wider network.

Of course this network can’t reach the internet because nothing knows how to get to the 192.116.200.2 address. We can solve that with some NATting in the root.

Since I have no iptable rules on this machine I can build a simple set. In my case br-lan is the bridge I have connected to my LAN network, so the rules would look something like

echo 1 > /proc/sys/net/ipv4/ip_forward

# Flush forward rules, policy DROP by default.
iptables -P FORWARD DROP
iptables -F FORWARD

# Flush nat rules.
iptables -t nat -F

# Enable masquerading of 192.168.200.0
iptables -t nat -A POSTROUTING -s 192.168.200.0/255.255.255.0 -o br-lan -j MASQUERADE

# Allow forwarding between br-lan and ip4only-root
iptables -A FORWARD -i br-lan -o ip4only-root -j ACCEPT
iptables -A FORWARD -o br-lan -i ip4only-root -j ACCEPT

And now, with all of that, the namespace can reach the internet

$ sudo ip netns exec ip4only ping -c 1 www.google.com
PING www.google.com (142.251.40.132) 56(84) bytes of data.
64 bytes from lga25s80-in-f4.1e100.net (142.251.40.132): icmp_seq=1 ttl=117 time=4.47 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.474/4.474/4.474/0.000 ms

If we try to use ipv6 then we get an error

$ sudo ip netns exec ip4only ping -6 -c 1 www.google.com
connect: Network is unreachable

Great, this is working!

Reducing permissions.

Unfortunately to enter a namespace with ip netns exec requires root and the command we run inside the namespace is run as root (we can see that from the prompt changing from $ to # in the earlier output).

Fortunately we don’t need root if we have the cap_sys_admin permission.

This is easily to do with a simple C program:

// ip4only
//   gcc -o ip4only ip4only.c
//   sudo setcap cap_sys_admin+ep ./ip4only
//
// Now we can do "ip4only command" (eg "ip4only ip addr")
// and it will run in the ip4only network namespace

// We need this for setns()
#define _GNU_SOURCE

#include <fcntl.h>
#include <sched.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>

#define errExit(msg) { perror(msg); exit(EXIT_FAILURE); }

int main(int argc, char *argv[])
{ 
  int fd;

  if (argc < 2)
  {
      fprintf(stderr, "%s cmd args...\n", argv[0]);
      exit(EXIT_FAILURE);
  }

  // To set a namespace we need to have an open file handle.
  // Network namespaces live in /var/run/netns so that's easy
  
  fd = open("/var/run/netns/ip4only", O_RDONLY);
  if (fd == -1)
    errExit("open");

  // Join the namespace
  if (setns(fd, 0) == -1)
    errExit("setns");

  // Run the specified command
  execvp(argv[1], &argv[1]);

  // If we got here, there's an error!
  errExit("execvp");
}

And that’s all there is to it. We can see it works by trying to access IPv6; we expect it to fail.

$ ip4only telnet -6 www.google.com 80
Trying 2607:f8b0:4006:820::2004...
telnet: connect to address 2607:f8b0:4006:820::2004: Network is unreachable

And it works.

I can now do ip4only speedtest and it now connects every time using IPv4

$ ip4only speedtest

   Speedtest by Ookla

      Server: DediPath - Secaucus, NJ (id: 22774)
         ISP: Verizon Fios

Conclusion

There is a side effect; “ping” no longer has the permissions it needs to run, and it gives a socket: Operation not permitted error. However if an application isn’t trying to use DGRAM or RAW sockets then this type of configuration is a nice way of using namespaces and capabilities to force the application into an IPv4 only setup.

ChatGPT and Calculators

Sat, 04 Mar 2023 20:30:23 -0500

This is one of my infrequent “philosophical” type posts. An earlier version of this appeared on LinkedIn.

There was a LinkedIn post along the lines of “are we treating ChatGPT today like we used to treat calculators in the past”.

In my mind the question is “what skill do we believe is valuable that ChatGPT will replace”.

The parallels between how we treated calculators in a school setting (“no you can’t use them for homework”) vs how we’re treating ChatGPT (“no you can’t use them for homework”) needs a deeper dive.

Calculators

40 years ago we believed mental arithmetic was important; calculators and now computers show that’s not true. Well, to an extent.

I grew up during that period; we were taught how to use calculators, but we weren’t allowed to take them into maths exams; we were expected to know basic rules and be able to derive solutions. We even had a book of common formulas and log tables and so on.

So it might be fair to ask “Do I need to know the derivative of sin(x^2)?“. 40 years ago I could do that using the Chain Rule. 30 years ago I had a calculator that could do it. Today? I’ve no idea. But I can google it… (oh, right; d/dx f(g(x)) = f'(g(x)) . g'(x) so the answer is cos(x^2).2x). I’ve never needed to know that since leaving education.

However, it might also be fair to ask “Do I need to know what is 7*12?“. Now I can also use a computer for this, but I do know the answer; it’s useful in “7 ft 5 inches is 89 inches” type manner; I do simple arithmetic all the time.

It’s also useful to act as a sanity check; if the computer does a bunch of calculations you may be able to do some rough approximations to determine if the answer is sane. Computers make mistakes. Just ask British Post Masters. People have gone to prison because of belief that the computer was right.

So we may have been right 40 years ago to have concerns around the use of calculators; or perhaps more accurately around the blind trust that the machine always gave the right result.

ChatGPT

What skill will ChatGPT replace? Is it a skill we consider important; or will it just lead to enforcing the real skill (critical thinking) vs the mundane (data collation)?

Now there’s also a qualitative difference between the two; 2+3=5 whether you do it by hand or by computer. But can we evaluate ChatGPT output the same way?

Given we know that current generation AI solutions give false answers very easily (see Google Bard’s demo day issues) there’s a definite need for critical thinking.

But without the knowledge base, how can you be critical? I can do a simple approximation in my head of a shopping bill and get something close to the total; “OK, the till receipt is probably accurate”. But how can I evaluate what an AI tells me about a topic I don’t know anything about?

Modern AI systems are kinda a black box; we train them with a bunch of data with known results, and the system learns to adjust variables so that it can better evaluate. Eventually the goal is that they can be fed new (known) data and come up with the right results; we then let it be used on unknown data.

The problem is that we don’t know what happens inside the box. A trial last decade on detecting skin cancer based on pictures had a good result on the training and validation data. But it turned out that the system was using the presence of a ruler to determine if a mark was cancerous or not. This is a great example of bias in the training data (pictures of cancerous marks were likely to be taken by a doctor and to have a ruler on it for sizing purposes) and led to the black box using totally the wrong data; of course it did well against the verification data!

Summary

Too many people believe “‘cos the computer said so” already (see the earlier post-office scandal). Add in some biases that may be encoded into the AI because of training data. Add in attackers deliberately tainting training data…

And I’m not going to even get into melt-downs, like a Microsoft AI chatbot going full racist after interacting on twitter. That’s a different problem entirely.

In my mind the challenge with a technology like ChatGPT is in how can we verify the output? We’ve developed tools against “Deep Fake” videos because there are discontinuities. But when the output can only be evaluated by a human? Hmm…

I foresee a variation of the email spam wars; spambots will do better to bypass filters; filters will get better to block the bots… rinse, repeat.

Who needs Tucker Carlson when the computer can make up the lies customised to your desires and you have no way of knowing if they’re true or false?

This is the challenge we have to face. That is the critical skill we need to retain.

Looking at YubiKey 5

Thu, 17 Nov 2022 08:53:43 -0500

I don’t normally write about specific products, but I was asked to take a look at the YubiKey series (primarily 4 and 5) and write up a summary of when and how it can be used.

This is timely, because CISA is pushing for access management enhancements and recently published a chart for phishing resistance.

I thought this interesting; typically I’ve looked at this from a user perspective (“can I use this to secure access to my bank account?”). But this time I needed to look at it from an Enterprise perspective (“Is it safe to allow our employees to use this to access our systems?”).

There’s a different set of challenges and requirements. As a user I could take a risk (“Eh, I can store the password in an encrypted text file and trust no one can access my machine”), but from an enterprise perspective we can’t necessarily trust the employee to make the right decision; they’re not security professionals and may expose the company to risk.

Lots of authenticators

YubiKey supports a number of authenticators, and they have different characteristics. Some of them are suitable for passwordless, others may only be suitable for MFA and others… probably shouldn’t be used, at least for workforce authentication, but may be good for customers (eg Harry logging into his bank account). I’m going to summarize as best I can. This has got a little long, but there is a table at the end!

Note that the YubiKey supports all of these at the same time. So we need to make sure we know what mode is being used at any time.

I might have got some things wrong. Let me know in the comments if you spot something!

U2F

U2F (Universal Second Factor) is a strong phishing resistant token.

When registering a device with a URL the YubiKey generates a key pair using the target protocol, hostname, part as part of the seed value (along with embedded encryption key that’s unique the YubiKey) to generate the keys. This means that abc.example.com will get a different keypair to abc.example.org. The public key (and “handle”) is sent to the remote server and that’s what is stored as your authentication token. This handle may include the private key encrypted with the encryption key embedded into the YubiKey.

When you then try to login to a site the protocol/hostname/port and handle are sent to the key. The YubiKey will verify the handle matches the protocol/hostname/port and was issued by this key. It will then use the decrypted private key to complete a standard public key authentication process. If a different key was used then the embedded information is different so the generated key would change, so the server can be sure the person has possession of the same key that was used for registration. Similarly if a different URL was used then the protocol/hostname/port would be different and so the handle wouldn’t validate; a MITM on a phishing URL wouldn’t be able to intercept the response and use that to login to a real site. MITM is possible at registration time, of course, so a user could think they were using secure MFA when they’re really not.

U2F may be implemented by software and so not be strong (encryption keys could be copied to multiple places, for example). There is an attestation process available, where the public key sent from the YubiKey is signed with the vendor attestation key. This means that it’s possible to verify that the U2F key is “strong”. Without this attestation we can not be sure the device presented is a hardware token or a piece of software with copied keys. With attestation it may be suitable as a 2nd factor, but without we can’t assume it’s strong. We may allow it for customer authentication without attestation; “If Harry wants to use a weak MFA device then it’s his lookout!”

FIDO2

FIDO2 is very similar to U2F. It’s an evolution of it, and the specs are backwardly compatible. However FIDO2 introduces some new functions. In particular it allows for forcing the user to prove they are the authorized holder of the device; typically this is a PIN or biometric.

By default YubiKeys do not protect FIDO tokens, but when the UV (User Verification) flag is set then the user will be asked to set a PIN or biometric. With YubiKey 4 the PIN is minimum 4 characters, with YubiKey 5 the PIN is minimum 6 characters. Note the PIN need not be just digits; any normal alphanumeric can be used. After 3 failed PIN attempts the device needs to be removed and reinserted. After 8 failed PIN attempts the token is locked. Unlocking requires a “master reset” and removes the private key, so effectively invalidates the token.

As with U2F there is an attestation process. With attestation and UV then this becomes a viable passwordless solution. With attestation but no UV then it may only be used as a 2nd factor. Without attestation at all then we may only use it for customer authentication, same as U2F.

Since the attestation and UV requirements are set by the server (on registration and login) it’s possible for different services to have different stances. This could get confusing. We can not simply say “FIDO2 is good for passwordless”; we must say “FIDO2 with device attestation and restriction to YubiKey and with User Verification is good”.

Microsoft Azure AD can perform FIDO2 authentication and appears to have the necessary configuration options

Windows desktops can be configured to support FIDO2 authentication, but the machines must be Azure AD (or hybrid) joined. Simple domain-joined machines can not do this. There are other limitations.

Apple have committed to supporting FIDO.

Non-web based applications can use FIDO2 via “out of band” mechanisms; e.g. the user may get sent a link (SMS, Email, copy’n’paste…) that they enter into a browser and complete authentication there. This obviously requires application changes.

Both Android and iOS devices also provide native FIDO2 authentication mechanisms. In testing on my Android device with the UV flag set I was prompted to fingerprint.

PIV (Smart Card)

In PIV mode, the YubiKey acts as a smart card. A public/private key is stored on the key as certificates. The certificate is signed by a CA trusted by the service. Essentially this follows a similar process to client-side authentication in mTLS; the server can validate the public key matches requirements (signed by a CA, contains the right metadata, etc) and then do standard public key cryptography to verify the private key is present.

YubiKeys automatically protect the certificates with a PIN (see FIDO2 about PINs); the PIN can not be removed (YubiKey bio devices don’t support PIV mode at present). The private key can not be extracted from the device. Because this is a certificate based process the keys can expire (and so need renewal) and a CRL (certificate revocation list) is available.

The key can be put onto the card in two different ways.

The first is that an external program can generate the key pair, get it signed by the CA, then put them on the card. This is problematic because it means the private key will live (for at least a small time) outside of the secure device and so could be copied. We also don’t necessarily have a way to verify the device is a trusted YubiKey and not an emulator.
The second is to have the YubiKey generate the key pair, and the CSR (certificate signing request). This generation process comes with an optional attestation process, which can be used to prove the keys were generated by a YubiKey and not a piece of software; the CA signing process can perform the validation and only sign the cert if it passes. The signed certificate can then be placed back on the device. In this way we can be sure that the private key was securely generated on the YubiKey and can not be copied.

We should require the YubiKey generation process be followed, and never store private keys outside of the secure device. When the key is used the user is prompted for their PIN/biometric. This gives us sufficient trust to let the YubiKey in PIV mode be used for passwordless access.

Windows supports PIV mode using the built in smart card services.

MacOS can support smart card logins, but on Intel Macs this may not work well with FileVault disk encryption since that needs the password to decrypt the disk. It’s possible m1/m2 based Macs solve this a different way. Macs may also have limited PIN support (numeric only).

Web authentication is also possible (most browsers support this) but the server needs to ensure the certificate is signed by the proper CA (same as with mTLS).

HOTP / TOTP / Static passwords

These three authenticators are lumped together since they use the same mechanism. Essentially the device acts as a USB keyboard and types the “secret”. This is very flexible. There is no PIN protection.

YubiKeys come with two slots. The first is accessed with a quick “short” press. The second requires a “long” press (maybe 3 seconds?). Since these are just “typing” they are not phishing resistant, and are just a “something you have” factor.

Each slot can be configured with HOTP/TOTP/Static credentials.

HOTP

“Out of the Box” slot 1 is configured with a Yubico managed HOTP seed. When you single press the device will “type” a one-time value that changes each time and adds the RETURN; eg

    cccccbbbubrtvbkjrrjhrkdgbcjfichivdihefvlujrt
    cccccbbbubrtrgndglnnltrcekljgnfibikhkrnvivrj

This value is documented, and Yubico provide an API for this built in seed.

The first part (cccccbbbubrt in this case) is the device identity; the remaining part of the value encodes things like “number of presses”.

Being HOTP the value changes each time you press, based on the number of times you’ve pressed. The validation server needs to keep track of the “last press” count for the device and only approve the login if the “new press” contains a count that is greater than the last value. This prevents replay attacks.

Using the built in seed it is possible to call the public Yubico API to validate the value. This is very simple, and even doable as a shell script.

It becomes very easy to create a self-service onboarding process. To register the device the user merely needs to tap the button (and so “type” the current code). The server can validate this (hence attest it is a YubiKey) via the Yubico API and save the device ID. For login the server simply needs to validate the code (again via the API) and verify the device ID matches the one saved.

This does have an external dependency on the Yubico servers being available in order to use this, but it provides a good and simple “out of the box” experience. You can use any off-the-shelf YubiKey (that supports HOTP) plug it in and self-register. We can be confident that it is a YubiKey being used; the call to the Yubico API verifies this is a YubiKey and not another device.

The seed can be replaced with a corporate owned one and you can run your own verification server, but then we have management complications (how to generate a unique seed for each device; how to ensure the device is a real YubiKey) so I don’t recommend this. I’d say that you should only do this if you’re worried about the Yubico server availability.

If the default seed is over-written then it can not be restored; only YubiKey can do this. If a “smart” (dumb) user uses the YubiKey manager (on a personal machine, for example) to delete the seed then they will no longer be able to use it for Yubico HOTP authentication.

TOTP

TOTP mode is similar to HOTP but uses “time based” rather than a count of touches. Since the YubiKey doesn’t have a built in clock it works in conjunction with the YubiKey Authenticator application. I think the authenticator app also stores an encrypted form of the TOTP seed (because of limited storage on the YubiKey, but I’m not sure of this). It will talk to the YubiKey and pass the time (and the encrypted seed? Not sure) to the YubiKey; the YubiKey performs the calculations and returns the current value to the app which the user can then type or copy/paste.

The problem with TOTP is in how the seed is copied; it’s generally exposed to the user (eg QR code) so they could screen shot it and add it to multiple authenticators. We also have no guarantee they are using a YubiKey and not some other program such as google authenticator. This makes it unsuitable for authentication for employees. Although I’ve seen similar mistakes with RSA SecurID softtokens…

Static Passwords

And finally a slot can be configured for static passwords. I guess if you have a 28 character password then it might make sense? Storing a password in this way changes the password from “something you know” to “something you have” because possession of the YubiKey is all you need to use the password. Unfortunately we can’t stop a YubiKey being used this way, but this isn’t a problem unique to these devices; any “auto-typing” HID device could do this. We just have to trust people aren’t knowledgeable enough (or stupid enough) to do this. One YubiKey uniqueness, though, is that because the same device can possess the password and MFA (eg HOTP in slot 1 and password in slot 2) then MFA may be reduced to a single “have” factor; “long press for the password, short press for the MFA”. Yet another reason to move to passwordless!

PGP

I include this for completeness, because YubiKeys can support it. PGP is a tool for signing/encrypting email. It requires a public/private key. Typically these are stored on the filesystem, protected with passphrases.

With a YubiKey the private key can be generated/stored on the device and are protected with a PIN similar to PIV mode. This makes it possible to have stronger protections on this key.

However, many organisations need to have an enterprise managed encryption solution (potentially with a corporate master key) for legal and compliance reasons. Individuals using PGP would break that.

It is possible to configure some ssh clients to use gpg-agent (instead of ssh-agent) and so effectively use the YubiKey as a store for your ssh private key but it’s not a supported configuration on all clients, and I don’t think humans should use ssh keys to access servers unless you have strong management processes in place.

So, really, we might not want use this functionality for authentication.

Summary

This got long, because a YubiKey can do so much! Here’s a quick table of my conclusions.

U2F
Phish Resistant	Y
Passwordless	N
PIN Protected	N
Attestable	Y
Authenticator Type	"Have"
Interfaces	Primarily Browser
Suitability	2nd factor if attested
FIDO 2
Phish Resistant	Y
Passwordless	Y*
PIN Protected	Y*
Attestable	Y
Authenticator Type	"Have" + "Know" (PIN)/"Are" (Biometric)
Interfaces	Browser, Desktop login
Suitability	2nd factor if attested; passwordless if attested and UV enforced
PIV (Smart Card)
Phish Resistant	Y
Passwordless	Y*
PIN Protected	Y
Attestable	Y
Authenticator Type	"Have" + "Know" (PIN)
Interfaces	Browser, Desktop login
Suitability	Passwordless if key generated on device
HOTP
Phish Resistant	N
Passwordless	N
PIN Protected	N
Attestable	Y*
Authenticator Type	"Have"
Interfaces	Keyboard typing
Suitability	2nd factor with Yubico default seed
TOTP
Phish Resistant	N
Passwordless	N
PIN Protected	N
Attestable	N
Authenticator Type	"Have"
Interfaces	Keyboard typing
Suitability	None
Static passwords
Phish Resistant	N
Passwordless	N
PIN Protected	N
Attestable	N
Authenticator Type	"Have"
Interfaces	Keyboard typing
Suitability	None
PGP
Phish Resistant	n/a
Passwordless	N
PIN Protected	Y
Attestable	n/a
Authenticator Type	"Have" + "Know" (PIN)
Interfaces	OpenPGP
Suitability	n/a

It’s important to note that this is just looking at what a YubiKey can provide. There are other solutions in this problem space, including applications on your mobile device that receive push notifications. Some of these just act as second factors, others can be completely passwordless.

However, sometimes you may not be able to use mobile devices (e.g. if you are in a call center where they’re not allowed in, or if you have employees who do not wish to install work related stuff on their personal devices).

In those scenarios a YubiKey or one of their competitors can provide a good strong solution. Just ensure you understand how to manage them and what their characteristics are.

Search This Site

Sat, 23 Jul 2022 11:26:31 -0400

Microservice Security

Sun, 26 Jun 2022 11:14:12 -0400

Recently I was invited to be part of a panel on Microservice Security. The fools! Normally on these panels they want you to talk for 5-ish minutes; unfortunately I came up with about 15 minutes worth of material!

That’s perfect for a blog :-)

Older designs

Before I talk about microservices I want to take a look at older designs

Monoliths.

A “monolith” is pretty much an “all in one” application. It may combine the UI, the app layer, the storage layer… all bundled together. I mean, who hasn’t written 10,000+ lines of spaghetti C code?

In a monolith we had data structures, but we didn’t really have much in the way of data segregation. Persistent data structures were pretty much global in scope. We could try and normalize access to these structures, e.g. using library functions, but the app couldn’t enforced this; instead we attempted to control things via code reviews… “why are you setting foo.bar rather than calling library function do_magic()?”

Now from a security perspective this wasn’t really too much of an issue; only the app internals could see it, anyway. Well, except it encouraged bad programming techniques and could have led to hidden bugs pretty easily, especially if modules got out of sync on the data structure formats!

Object Oriented Programming

This introduced the idea of public and private elements. Now we can hide things and enforce access through “approved” public interfaces, or methods. You can no longer set foo.bar directly, but instead have to call Foo.do_magic(). This can definitely help ensure data consistency and reduce bugs.

However, those public methods are, well, public. If I can see the object then I can call those methods.

But it’s still not really a security issue, for the same reason; only the app could see those methods.

Three tier apps

Now we’re starting to separate components. No longer do we have a monolith, and what used to be methods internal to the app may be exposed to the wider network. This brings in a set of security challenges.

For example:

How to we ensure the app tier is only called from the correct display tier?
How to ensure the DB is only reached from the app?
How to stop the expected data flow from being bypassed?

We might look at firewalls to do some of this. If you can’t see the service endpoint then you can’t call the functions. However this is limited and gets hard to manage at scale. Unless your environment has been designed from day 1 for micro-segmentation and defined communication paths you probably have a relatively standard “open” 3 tier network.

So the common solution is to require authentication between the tiers; e.g. the web tiers authenticates to the app tier; the app tier authenticates to the database.

And remember that the network may also need to be protected; data in transit may need to be encrypted if it contains sensitive components. Yes, even inside your own datacenter!

Multi-component apps

Docker was a big driver of this, but it’s also used elsewhere. The idea is to allow component re-use in a simple “Lego build” design.

“OK, let’s spin up an app with a memcached container, oh… let’s also use mysql, nginx, redis, mosquitto… and 6 different containers for my app.”

This is really a generalized case of the 3-tier app. So we can use the same type of solution.

Authentication; each component must know that the call has come from an authorized source
Encryption in transit
And let’s not forget input validation!

Instead of just doing this in a couple of places we might need to do this in a dozen places. This is getting complicated…

Finally, micro-services

Really, a micro-service design is just a generalized form of multi-component apps

Instead of monolith exposing functions to other parts of the application, each micro-service dedicated to a particular function or group of functions. And these functions may be exposed to the network!

NOW we care about the security of the function. They can be reached from outside of the application.

In some cases this exposure is deliberate. These functions should be considered “public”. They’re meant to be the interface between the application and the application clients (whether they’re internal to your organisation or external to the org doesn’t really matter; they’re external to the application).

In other cases this expose must be prevented. These functions should be considered “private”. But are they?

This public/private terminology is similar to that for Object Oriented Programming. It means public or private to the application… however that is defined. I am NOT talking about “private” to the organisation. Think zero trust :-)

How do we ensure these private interfaces are kept private? How do we ensure that public interfaces are only used by authorized consumers?

And we have the same answers.. authentication, encryption, input validation…

Well, we can do better…

All that is hard work.

Fortunately modern technologies can assist in newer designs that can help enforce the separation between public and private endpoints

Some examples of this include

Private network structures (eg docker-compose) so that the backend components (memcached, etc) aren’t exposed off the local machine; for single-app machines this may be sufficient to make an endpoint private. Docker swarm took that to a multi-machine level but required oversight for multi-app.
k8s; containers inside a pod can talk to each other. Great for private communication. Containers in other pods have to talk via public interfaces. Need to balance scalability of individual micro-services to pod-level scalability and easier communication…
Container network constructs; e.g. istio can act as a gateway between pods that will transparently do TLS encryption and can do mTLS authentication; set the rules as to what pods can talk to what other pods and it’s enforced by the infrastructure.

How you deploy stuff can answer some of the security concerns raised by going to micro-services.

Some other complications

These aren’t necessarily security issues, but to be kept in mind

Every public endpoint needs to be considered an API to your application. It defines a contract to your consumers. If you change your call (data structure, functionality, parameters, whatever) you need to treat it as an API change.
Change control gets harder; should each microservice have its own SDLC lifecycle? Be registered in your app inventory?
Knowing how to “right size” your microservices is a skill. If you go too granular then you get a lot of overhead (even if it’s just communication overhead! A network call is orders of magnitude slower than a local app function call) and lose performance and may have more paperwork. If you go too coarse then then you may not get the independent scalability you desire.

Summary

Micro-services require you to think about what endpoints are considered public and what are private.
Any network port that is exposed “off” the machine is automatically public. Even those only on localhost may be public on a multi-app machine
Authentication, Authorization, Encryption, AUDITING… these need to be enforced at every exposed endpoint! SDLC processes should be leveraged to help validate this.
Remember, data flows can be initiated differently from what your app expects; you may expect calls A->B->C->D but an outsider might call C directly if it’s public; you need to handle that.

This is a lot to keep in mind; not only must the code be written, but knowledge of networking, scaling, performance, encryption, authentication…

This really highlights the importance of SecDevSecOpsSec. A proper architecture design at the beginning can simplify the solution and take some load off the development team.

Secure your cloud

Fri, 04 Mar 2022 06:48:32 -0500

I got asked another question. I’m going to paraphrase the question for this blog entry.

Given the Russian invasion of Ukraine and the response of other nations (sanctions, asset confiscation, withdrawal of services, isolation of the Russian banking system…) there is a chance of enhanced cyber attacks against Western banking infrastructure in retaliation. How can we be 100% sure our cloud environments are secure from this?

Firstly, I want to dispel the “100%” myth. No security is 100%. Your on-prem environment isn’t 100% secure. The cost would become prohibitive and potentially greater than the value of the asset’s you’re protecting. Instead you perform an analysis and reduce the risks to a level you are comfortable with, at a price you’re willing to pay. These risk reduction strategies may include preventative controls, detective controls, impact mitigation strategies and more.

So the question, really, is “Are we sure our controls and strategies are good enough”.

Now there hasn’t yet been any confirmed reports of high threat attacks against financial institutions (going by the CISA reports), the UK GCHQ held a rountable to discuss the threat to critical infrastructure.

So what makes the cloud different? That’s a massive topic because “cloud” is too generic a term. I’m not going to talk about SaaS offerings like Workday, Salesforce and the like (although these are important and may contain sensitive data, so make sure they are controlled!).

And that includes O365! If you’ve outsourced your Exchange environment to Microsoft (and who hasn’t? It’s convenient) this could be a critical dependency… can you still operate if your email is down? Your Teams chat is down? Your very phone system (integrated into Teams) is down?

But that’s not where I’m going; Instead I’ll focus on primary cloud application environments such as AWS, Azure or GCP.

Shadow IT

Infrastructure sitting on-prem automatically gains a level of protection by being behind corporate firewalls. It can’t be reached from the outside (“you can’t hack what you can’t reach”) and it shouldn’t be able to make outgoing connections; firewalls control egress as well as ingress!

And your processes better require CMDB registration before firewalls will be opened (“no you can’t talk to the database from the app tier; firewall blocked!“; “no you can’t talk to the web proxy!”).

Even if the server isn’t in your CMDB it gains these protections.

But cloud Shadow IT… now a whole environment could be spun and exposed to the internet. This may not (should not!) have full access to on-prem resources and so may have a limited blast radius, but it may receive a data feed from on-prem services and so store or process data sensitive data.

And these environments won’t gain any of the traditional on-prem protections. They’re open, exposed… and we may not even know they’re there!

There are services you can sign up for that scan the internet and try to work out the owner (eg based on the TLS certificate information… if it has an OU=yourbank then it may assign that to YourBank). They’re not always very accurate initially, and so have a cleanup cost. But once it’s clean any new “Hey, we just found this!” change should trigger a review… is it another false-positive or did someone spin up shadow IT?

Borders

Which leads into knowing your borders. Again, on-prem datacenters have traditionally been built with a “hard shell and soft chewy center”. We have a well defined border. But with the cloud… ah, there’s a lot more borders than you may think. And they may change! Last year I even showed that Alexa could be used to exfiltrate data!.

This may be the worst part of the cloud; stuff is now exposed to the internet that, really, shouldn’t be. This may include data stores, control planes and more. Since these things may be exposed to the hostile internet, access control, auditing and monitoring become critical

Data Stores

Here’s another area where the cloud is different. Once again, traditional on-prem stores gain a level of protection just by being on-prem. The cloud… this may expose your data stores directly to the internet! Indeed a number of years ago there was almost a new breach notification every week because someone had left data in an open S3 bucket. Yay.

Cloud service providers (CSP) have matured since then and a number of these services now have private endpoints that are exposed only to your VPC, but you need to verify that every data store you have configured (S3, RDS, DynamoDB…) are all set up securely. If any are exposed to the internet then you need to verify they are properly restricted via IAM policies!

Another area the cloud is different at is activity monitoring; your tradition DAM (Database Activity Monitoring) solution may not work, because it may require an agent installation on the server or similar. We can be limited by what the CSPs service provides… and this is variable and different across CSPs, and even different between offerings within a CSP! If there is a breach (eg bad IAM policies, or a pivot from a trusted server) can you track what data the attacker has seen?

But it’s not all complications. Just by being separated from the primary desktop environment, cloud data stores are less susceptible to things like ransomware or wiperware. These typically get into an environment via the desktop, and local disks and NAS mounted volumes may quickly be encrypted or data erased. Your cloud block storage and cloud databases are sufficiently removed. These type of attacks are a common tactic, and the CISA are claiming that Sandworm (aka Fancy Bear), suspected of being associated with the GRU, have new malware. This organisation has a history of ransomware and wiperware, including NotPetya.

Control Plane

And here is where we start to wince. The control plane is possibly the most sensitive part of an infrastructure; you can create VMs, destroy VMs, look at their data, modify permissions… lots of stuff.

Your on-prem control plane may even be on a separate network segment because it’s so sensitive. But the cloud one? That’s exposed to the internet and can not be changed. It’s designed to be accessible everywhere.

The best you can do, here, is to ensure your access control policies are tightly locked down, and auditing and monitoring is in place.

IAM

For a lot of CSP native services, IAM is the core of the access control. It controls everything. Who can access what. What resources are exposed. Where access is allowed from.

If you mess up your IAM policies, you’re asking for a world of hurt.

And it’s not simple. Looking at an AWS IAM policy is like staring into the abyss; do it too long and it’ll stare back at you and you’ll start seeing the whole world as a JSON policy!

Fortunately tooling now exists for this. Even better, the API nature of the cloud means that access need not be so static. We can move from the traditional “minimum necessary access” through “just in time access” all the way to “zero standing access”. You want access to the cloud control plane? Jump through these hoops, do some MFA, and you’ll get the access you need… and we’ll remove it in 2 hours time. In AWS you don’t even need an account; just assume a role.

As with networking “you can’t attack what you can’t reach”, you can’t login to accounts without access.

This tooling can also help with IAM role analysis; are people over permissioned? Are people granted permissions who never use the account? Can we define RBAC models?

I don’t expect many companies are at this stage, today (when we’re supposedly worried), but it’s a target they should be looking at.

But even without this zero standing access control, we already know how to control IAM access, monitor it. It’s a critical area, considering what it controls (including the control plane!) so we’ve put controls in place.

Network Segmentation

Now here is another area the cloud can be better than on-prem. Because of the Infrastructure as Code (IaC) nature of a cloud environment it’s very easy to create macro segmentation between applications. Instead of the nice chewy center of your traditional datacenter we now have a honey comb. East-West traffic can be controlled.

This helps control the blast radius if an application is compromised (again, you can’t attack what you can’t reach). But, further, it makes it easier to shut down access. If an application with access into your core services (e.g. the mainframe) gets compromised it is easy to block all traffic to/from that application. We’ve now contained the problem, prevented further infiltration into the environment and blocked further data exfiltration.

The out-of-band nature of the control plane also now means you can easily snapshot the service and preserve it for forensic analysis.

The cloud can make the whole DFIR process easier than on-prem. Suspicious behaviour can even pre-emptively start snapshotting and tracking of activity before a breach occurs.

Modern security tooling

The cloud makes it possible to do security in new ways.

For a traditional VM in your datacenter you need to do a credentialed scan of the environment to see what’s really going on. This may be remote initiated (eg via ssh), or using an agent. And now we have to worry about coverage (are my agents on every machine? Are they working?). Which requires an accurate CMDB. How frequently is that updated? Are decomm’d assets removed? What about machines that are powered down?

The cloud, though… it can tell you an accurate “as of now” inventory of resources you have, and their state. And you can snapshot the disks (whether the instance is running or not) and scan the snapshot for malware or insecure software or dodgy entries in log files or…

Spin up a new VM and you get scanned even without needing to worry about agents!

Add in information from network flow logs and you can start to build connectivity graphs between assets, and help determine actual risk to a vulnerability.

Of course it’s not a panacea. They can’t see inside the running box, so fileless malware won’t be detected. They also can’t see images from filesystems they don’t understand. They can’t read locally encrypted disks (eg via dm-crypt). And they can’t see process execution. You still need agent/credentialed scans for that. The best of both worlds is using the “ambient” security at the CSP layer, enhanced with data from agents.

But just having the ability to scan something because it is there is a massive change in security. It’s now “ambient” to the environment; something that just happens with no additional effort.

CVSS isn’t necessarily good.

Many risk controls are based around a CVE. “Oh no; this has a 9.5 rating! Fix now!“. But this isn’t necessarily an accurate definition of your actual risk. “Yes, log4j is sitting on this box, in a directory called /opt/myapp_oldversion with a permission of 000.” But I’m not at risk.

This new tooling now lets us look at vulnerabilities in a different way. Let’s say we have a Apache RCE. We find this on 10,000 machines. Do we need to patch them all ASAP? Well, we can look at the risk. Internet exposed services are clearly “fix yesterday”. Non-internet facing is still at risk (eg insider attack) but could be lowered in priority.

But how do we know if something is internet exposed? The connectivity graph can help here; “yep, this goes through a firewall, through a WAF, through a load balancer… even though it only has an internal IP address, it’s internet exposed!”

I’m not sure any tooling is really that good, at the moment, and even the modern tools can’t see inside non-native configurations (e.g. if you have a Palo Alto virtual appliance then the tooling has no idea about connectivity). So our scope to automate this is more limited.

But even if we do come up with a good risk score for each instance based on the best data, is this the right thing?

There’s a number of known vulnerabilities without a CVE at all. And, of course, there’s always a zero day.

Using CVEs and risk scores to target remediation activities and thus lower your risk is good. But it’s not, in my opinion, sufficient. It’s just another part of a defence-in-depth strategy, along with firewalls, WAFs, auditing and the rest.

Your threat intelligence feeds tell you that there’s a new attack on a component; your ambient security tools tell you we have a lot of vulnerable components; we program the WAF to block traffic matching detected patterns and then use risk scoring to prioritize remediation.

I would argue that reporting risk based purely on CVSS scores is doing your organisation a dis-service.

Are we there yet? Probably not! But the new tools I have at my disposal give me more visibility into my cloud environment than I’ve ever had on-prem!

Attacks on the CSP itself

Of course this has all been looking at the attack as an attack on the company.

We also need to consider attacks on the CSP. If AWS infrastructure gets breached then all of the controls I mentioned above don’t necessarily mean a thing.

This isn’t an idle thought, either. Last year Azure CosmosDB had a vulnerability (ChaosDB) which could have exposed data for years. Azure container instances had a flaw that could have led to cluster admin permissions. And, Azure Functions had an exploit.

To the best of my knowledge, all potential CSP exploits were responsibly disclosed and fixed before data had been exposed.

But what if our Russian friends found an AWS zero-day? What is our exposure?

This isn’t easily measurable. The CSPs, themselves, have cyber teams to detect actual exploit and infiltration. What we’re doing as the customer of the CSP, they’re also doing on the underlying infrastructure.

Is it 100% perfect? Obviously not.

CSPs are a high value target for attackers; break into that and you have potential access to thousands of victims. But they’re not the only people in this privilege position. Traditional MSPs have, previously, been used as a route to attack their customers. Every VPN you have to a provider, a partner, a client is an attack vector. Even traditional on-prem equipment (eg firewalls) have had vulnerabilities.

Conclusion

In some places cloud is better and new technology solutions can make security easier and more pervasive than traditional on-prem solutions. In other places we’ve opened new borders, lost traditional protections, exposed critical interfaces to the internet; these all need solutions that on-prem didn’t need because the risk profile is now different.

The security controls and processes are different as a result.

But the cloud isn’t new. I’ve been working with the public cloud for 6 years and I’m a relative newbie compared to many. These are all known problems and you should have solutions already in place; even if they’re just detective.

I suspect most breaches will occur at the application layer, and that’s the same on-prem or in the cloud.

So am I 100% confident our cloud environments are secure from a state sponsored cyber attack? No. But I do have an equivalent level of confidence as I do for on-prem environments!

The problems with port 80

Wed, 02 Mar 2022 20:55:02 -0500

I got asked a question… this gives me a chance to write an opinion. I have lots of them!

If I redirect my port 80 traffic to another site, do I need to get a TLS cert?

The question here is related to if a bank (or other service) has changed their name, then do they still need to maintain a TLS site for the old name? Can’t they just have http://mybank.com send a redirect to https://newbankname.com and be done with it?

Why do we use TLS?

There are a number of reasons that we use TLS connections over the internet (including, but not limited to):

Confidentiality - the data transferred can’t be seen by any person in the middle because it’s encrypted
Identity - the TLS certificate tells you that you are talking the website you think you are
Integrity - the data hasn’t changed in transit.

All of these are good security features. Even supposed “good guys” may abuse non-TLS connections. For example, in 2015, AT&T inserted adverts into traffic through their hotspots. In 2014 Verizon added tracking cookies to their mobile traffic. Both of these took advantage of the lack of Integrity and Confidentiality in the non-TLS connections.

Stub server

So, typically, we don’t serve websites over HTTP on port 80, but use HTTPS on port 443. However there is sometimes a “stub” left on port 80 to redirect. And this stub isn’t always that good.

For example, the browser might say “Hi, I want to go to http://mybank.com”, and it will make the port 80 connection, and the server will say “OK, really, go to https://www.mybank.com” or sometimes even “Yeah, we’ve changed names; go to https://www.newbank.com”. And now all the traffic is on port 443 and we’re good. Right? Well, not so much…

Where can this go wrong?

An attacker can look at that initial port 80 request, and change it. Let’s look at those three reasons listed above and see how the bad guy can abuse this

Confidentiality - this isn’t so big a deal. The initial request your browser sends to the server and the response might be visible to the attacker, but they can’t see anything beyond that.
Identity - now we’re getting somewhere. With HTTP you don’t know the server you’re talking to is the server you think you’re talking to. The attacker could have redirected traffic to their own machine. Now when you ask for http://mybank.com your browser may get the response “OK, go to https://mybankk.com”. Notice the extra k? If you’re not observant then you may not notice anything different. It’ll even be over TLS with a nice certificate, for mybankk.com, so your browser happily puts up the lock icon. So now you’ll enter your bank login details onto a rogue server… and the bad guy steals all your money.
Integrity - here’s also a fun one. Your browser sends the initial request and the server sends the response, but the bad guy changes the information so that it, again, reads mybankk. As far as the browser is concerned that’s a good reply and you’ll be redirected to the bad site. (This is basically how AT&T injected their ads, how Verizon added tracking).

How can these attacks happen?

It depends on your situation. Let’s say you’re sitting in Starbucks using their free WiFi… well, are you? Or are you using an attacker’s hotspot that looks like Starbucks, but isn’t. They can steal your unencrypted traffic and so redirect you to their bad site. If you’re at home, is your home router secure? Or has it been broken into? A number of home routers have known weaknesses (backdoors, default passwords, etc) which could let an attacker perform that integrity attack. Or even if your home router is good or you are on a good Starbucks server, the attack may have been able to steal network routes (BGP attack) or hijack DNS so that your browser goes to the bad server. There will be other attack vectors; I just listed a few.

I’ve previously written some blog posts on this:

These aren’t attacks on you; they just looking for traffic going to mybank.com. You just end up being a victim.

So what can we, as the website owner, do about this?

Firstly, if you are using a domain (e.g. www.mybank.com) then you MUST have a TLS version of the site available. This TLS version must set the Strict-Transport-Security header on the https connection. Now, after the first time a person goes to a website the browser will remember “Hey, I should always go to the HTTPS version” and will not try port 80 again. This really reduces the scope of the attacks listed because if you’ve already visited the bank before then your browser already knows to avoid port 80.

BUT: This only works for the same domain name (and potentially subdomains), so if you have https://newbank.com set the header then this will not protect http://mybank.com because it’s a different name. That means that every name needs to have the https version of the name also created (with its own Strict-Transport-Security header), and so have a TLS certificate. Without that the browser will always go to the port 80 version of the site when requested, and be subject to the attack scenarios listed.

Even if you, yourself, do not run a port 80 version of your website you still must have the 443 TLS version with this header, because the previously discussed bad guy doesn’t need you to have port 80 open; they can direct traffic to their own server!

Corporate Application Security standards should require the header to be in place and check for it with DAST scans. If you’re not checking this header as a matter of course then start doing so!

Now make sure any redirect you do from http to https stays within the domain. So if you want to go from http://www.mybank.com to https://www.newbank.com then you need two steps:

http://www.mybank.com to https://www.mybank.com
https://www.mybank.com to https://www.newbank.com

If you miss that middle step then the Strict-Transport-Security header won’t protect the initial site!

If you want to get more advanced, once you have the TLS certs in place then you can request it gets preloaded into Chrome and other browsers. Basically your URL gets encoded into the browser so when you go to a site then the browser will always try https!

Note that some browsers are experimenting with “https first” designs to help protect people from these types of problems. However, as the website owner you can’t rely on this.

Summary

So to cut a long story short (too late!) that simple redirect on port 80 may be causing more headaches than you think. It definitely opens up some vulnerabilities which, if exploited, would probably get yourself into the press. The mitigation listed (Strict-Transport-Security) requires a TLS certificate for each domain. Even if you close port 80 you’ll still need to have a TLS server on 443.

Yes, keeping www.mybank.com open to redirect to www.newbank.com turns out to be more complicated than just a simple redirect!

As an aside, from a pure market perspective, if I ever saw my bank do a redirect on port 80 to another domain then I’d close my account ASAP!

Singe cloud or multi cloud?

Tue, 15 Feb 2022 17:52:29 -0500

I got asked a question… this gives me a chance to write an opinion. I have lots of them!

Is it reasonable to just stick with a single cloud provider, or is it better to go multi-cloud?

It think it seems reasonable. I expect very few places are true multi-cloud, as in a given app runs in two clouds. That becomes challenging if trying to use cloud native services ‘cos how you access RDS would be different to how you access Azure SQL, so writing a true multi-cloud application isn’t so simple. Especially when you start to do Infrastructure as Code (IaC) where your Terraform templates may be totally different; “why does this work in AWS but not Azure?!”

And if you try to avoid that by basically self-hosting everything (just using the IaaS) then you avoid many of the benefits of cloud (you’re just outsourcing your datacenter) and might as well stick with a provider of choice, for simplicity.

Businesses that are multi-cloud would tend to have App1 in AWS, App2 in Azure, App3 in GCP…

We’ve made it simple for app teams (they just target a simple cloud) but this becomes a headache to secure because structures work differently between CSPs; IAM processes are different, auditing and reporting is different, firewalls/WAFs are different. Even things which may seem simple are different. Even VPC can mean different things between providers!

It’s not to say you can’t go multi-cloud for a single app (and handle data gravity issues and so on), but my feeling is that these are more the exception than the rule.

Every cloud provider you use may require a whole new set of tooling to manage it.

Note: this ignores SaaS offerings like Microsoft Office 365, or Workday or and the like. I suspect everyone is multi-cloud when you take SaaS into account :-)

what about a hybrid cloud with a solution that bridges into public cloud providers so that they are basically one cloud everywhere, even on-prem. Isn’t that insecure?

Just because there’s connectivity, it doesn’t mean that there are no blast radius limitations. After all, even for pure public cloud hosted apps you’re going to need connectivity back to on-prem (eg via ExpressRoute or DirectConnect or whatever) just because your cloud-hosted app will need to talk to on-prem resources to function. Your core processing and orchestration systems may be on-prem and your app may have dependencies on them. Until you go 100% cloud there will be a requirement for communication between on-prem and the cloud.

How you limit your blast radius will depend on how far you want to go.

For example, in AWS, you could use VPCs as a form of macro-segmentation, and have an app per VPC. For traffic to leave a VPC it has to traverse a firewall where it is inspected and only authorised destinations (whether that’s another VPC or on-prem or even the mainframe). Azure provides similar constructs. You could consider this a “hub and spoke” model; all communication between spokes has to go via the hub and this becomes a good control point.

This isn’t micro-segmentation, but it’s probably better than most traditional datacenters which tend to focus on North-South traffic (web->app->DB) rather than East-West (app->app). The IaC nature of clouds allows for you to programmatically control both types of traffic!

With the right type of control plane I can see creating an app environment on-prem (with VMware) or in AWS or in Azure the same way and defining the ingress/egress controls to limit their exposure. Essentially you can treat the CSP as a separate availability zone, potentially with custom resources. This is what PaaS (eg CloudFoundry) promised.

So you breach into my b0rken Apache Struts app… you can’t get to Jenkins to infiltrate my CI/CD pipeline (hi, Solarwinds!) ‘cos you don’t have connectivity!

Of course you still need your traditional border controls, such as WAF, IDS/IPS, DLP between your cloud hosted environment and the internet.

A breach at the CSP level gets interesting, but the attacker then has different challenges. They might be able to jump into a VM by attacking the hypervisor, but then they still have the traffic restrictions. If they try to bypass the firewalls then they’ve got to somehow inject into the overlay network constructs… not practical!

Summary

Done correctly (ah, that’s the challenge!) a hybrid cloud need not be insecure. The risk is equivalent (if not less) than your traditional zoned datacenter. Especially if you are just single-cloud; again multi-cloud means you have to build these controls multiple times in different ways!

Of course I’ve just looked at this from an InfoSec perspective. There are other business reasons to look at multi-cloud.

What happens if your cloud providers decides to compete in your market space (Amazon are looking at providing financial services; Walmart won’t work with AWS)?

What happens if your cloud provider decides it’s not financial viable to provide a service due to losses?

What if your provider has a security issue and an attacker exploits it (fortunately didn’t happen here)?

From a security perspective, I wish I was single-cloud! Then I could focus on that. Multi-cloud makes everything much harder.

Hey, I said I had opinions!

Using SSH certificates - revisited

Sun, 06 Feb 2022 15:14:30 -0500

A while back I wrote about some basic usage of SSH certificates as an authentication system. I only described the core, but the comments went into some further detail.

I thought it time to write a follow up post describing some of the more advanced features.

Quick recap

To handle cert based authentication you need a CA certificate. This is created with the ssh-keygen command.

e.g.

$ mkdir ssh-ca
$ cd ssh-ca
$ ssh-keygen -f server_ca

Server certificates are similarly signed with the same command.

e.g.

$ sudo ssh-keygen -s server_ca -I key_for_test1 -h -n test1.spuddy.org -V +52w /etc/ssh/ssh_host_rsa_key.pub

More details on this are available at the original post.

Checking the certificate details.

We can look at the certificate, again with ssh-keygen

$ ssh-keygen -L -f /etc/ssh/ssh_host_rsa_key-cert.pub
/etc/ssh/ssh_host_rsa_key-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com host certificate
        Public key: RSA-CERT SHA256:ggTP5Bg7d74kmK+g+g8KVNizhmEz7ilCmO41GPRRVag
        Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
        Key ID: "key_for_test1"
        Serial: 0
        Valid: from 2022-02-06T15:08:00 to 2023-02-05T15:09:50
        Principals: 
                test1.spuddy.org
        Critical Options: (none)
        Extensions: (none)

We can see the Key ID values matches the -I parameter used on the signing command, the principal name is the hostname, and the validity period is 52 weeks.

User certificates

This is where we start to go deeper.

Let’s sign a user key the same way as we did before, and then look at it.

$ ssh-keygen -s ssh-ca/server_ca -I user_sweh -n sweh -V +52w .ssh/id_rsa.pub
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh valid from 2022-02-06T15:37:00 to 2023-02-05T15:38:31

$ ssh-keygen -L -f .ssh/id_rsa-cert.pub
.ssh/id_rsa-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:W+cq9v38nF6h1Wlse45h0qizmybkkj0J4Yh820ihG7A
        Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
        Key ID: "user_sweh"
        Serial: 0
        Valid: from 2022-02-06T15:37:00 to 2023-02-05T15:38:31
        Principals: 
                sweh
        Critical Options: (none)
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

It’s interesting to note that “Principals” implies we can list more than one principal (the -n option). “Extensions” is also interesting.

Principals

Generating a certificate with multiple principals.

This is done by specifying each principal separated by a comma. In a general corporate environment, with Active Directory, it might be worth generating certificates with the sAMAccountName and UserPrincipalName values. Typically that’ll be a short name and an email address. So, for example,

$ ssh-keygen -s ssh-ca/server_ca -I user_sweh -n sweh,sweh@sweharris.org -V +52w .ssh/id_rsa.pub
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh,sweh@sweharris.org valid from 2022-02-06T15:46:00 to 2023-02-05T15:47:10

$ ssh-keygen -L -f .ssh/id_rsa-cert.pub
.ssh/id_rsa-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:W+cq9v38nF6h1Wlse45h0qizmybkkj0J4Yh820ihG7A
        Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
        Key ID: "user_sweh"
        Serial: 0
        Valid: from 2022-02-06T15:46:00 to 2023-02-05T15:47:10
        Principals: 
                sweh
                sweh@sweharris.org
        Critical Options: (none)
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Using principals

In the default case, sshd tries to match the principal name to the username being logged into. This really works well most of the time; if I want to login as sweh and my certificate has sweh as the principal then it’ll let me in.

This is not so good if you’re trying to access a system, service, or shared account. e.g. root or (on AWS) ec2-user, or oracle or…

Side note: I don’t generally recommend logging in as these types of account directly. I prefer people to login as themselves and then use privilege escalation tools such as sudo to provide a stronger audit trail of who has done what. If 3 people have all logged in as root and the server gets rebooted, how do you know who did it?

Here is where we can use an option in sshd_config that’s normally disabled; AuthorizedPrincipalsFile

e.g.

AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

Here we can list the principals allow to access an account:

$ sudo cat /etc/ssh/auth_principals/ec2-user
testuser@sweharris.org
sweh@sweharris.org
$ ssh ec2-user@test1.spuddy.org id
uid=1000(ec2-user) gid=1000(ec2-user) groups=1000(ec2-user)

and looking at the logs…

Feb  6 16:09:23 test1 sshd[8160]: Accepted publickey for ec2-user from 10.0.0.158 port 34974 ssh2: RSA-CERT ID user_sweh (serial 0) CA RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU

Roles as principals

This gets hard to manage at scale. If you have thousands of machines then you may not want to list every possible sysadmin on every machine. Instead you could create a role name, such as sysadmin and your signing tool can add this to the to user certificate. eg sweh,sweh@sweharris.org,sysadmin. Now the AuthorizedPrincipalsFile only needs to list sysadmin and anyone who has that listed will be able to have access.

It’s recommended that if you use role based principals then the validity is reduced (eg 90 days) to help enforce that when someone transfers to a new department then their roles will be removed within a reasonable period. Using @revoked and maintaining a CRL may also be useful to ensure certificates can no longer be used.

More dynamic

A larger enterprise may wish to use AuthorizedPrincipalsCommand instead of a fixed file. This could do a dynamic lookup (e.g. in LDAP or an API call) to generate a list of acceptable principals. This could allow for real-time evaluation (“Who is allowed to login as ec2-user on host test1?“). Such a process would handle leaver/transfer handling (“Stephen is no longer a sysadmin; Fred left the company; John has become a DBA”) without needing to revoke certificates or require re-signed certificates.

Extensions

Thus far we’ve just talked about who can login to a server as a specific user (“Stephen can login as ec2-user”). With extensions we can be a little more fine grained.

By default the following are permitted:

        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

This can be changed on the ssh-keygen line with the -O option. e.g. to remove all the “permit” options can we can use the clear value

$ ssh-keygen -s ssh-ca/server_ca -O clear -I user_sweh -n sweh,sweh@sweharris.org -V +52w .ssh/id_rsa.pub
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh,sweh@sweharris.org valid from 2022-02-06T16:44:00 to 2023-02-05T16:45:52
$ ssh-keygen -L -f .ssh/id_rsa-cert.pub
.ssh/id_rsa-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:W+cq9v38nF6h1Wlse45h0qizmybkkj0J4Yh820ihG7A
        Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
        Key ID: "user_sweh"
        Serial: 0
        Valid: from 2022-02-06T16:44:00 to 2023-02-05T16:45:52
        Principals: 
                sweh
                sweh@sweharris.org
        Critical Options: (none)
        Extensions: (none)

We can also specify things like force-command (matching what we can put into authorized_keys).

$ ssh-keygen -s ssh-ca/server_ca -O clear -O force-command="/usr/bin/id" -I user_sweh -n sweh,sweh@sweharris.org -V +52w .ssh/id_rsa.pub
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh,sweh@sweharris.org valid from 2022-02-06T16:47:00 to 2023-02-05T16:48:06
$ ssh-keygen -L -f .ssh/id_rsa-cert.pub                                       
.ssh/id_rsa-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:W+cq9v38nF6h1Wlse45h0qizmybkkj0J4Yh820ihG7A
        Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
        Key ID: "user_sweh"
        Serial: 0
        Valid: from 2022-02-06T16:47:00 to 2023-02-05T16:48:06
        Principals: 
                sweh
                sweh@sweharris.org
        Critical Options: 
                force-command /usr/bin/id
        Extensions: (none)

Now if I try to use this certificate I can only run the one command

$ ssh ec2-user@test1.spuddy.org cat /etc/passwd
uid=1000(ec2-user) gid=1000(ec2-user) groups=1000(ec2-user)

These extensions can let us lock down certificates pretty much as tightly as the older private keys (command and from restrictions in authorized_keys).

The force-command and source-address options are particularly useful when performing app-to-app type communication. These keys generally don’t have a passphrase on them (since it’s automated) so have the ability to let one server scp a file to another without granting full shell access and enforcing the server the connection came from is important.

In a firewalled environment it’s worth thinking about always using -O clear to start with to prevent tunneling through firewalls. I would also add -O no-user-rc since ~/.ssh/rc can be used to bypass restrictions.

Conclusion

SSH certificates have a lot of power to them.

However they do require a good admin tool to manage them; to do the signing, to add roles (if used), to add restrictions, remove permissions, revoke keys and so on. I try not to recommend tools in this blog (so don’t ask!) but if you search the internet you’ll find a number of them, both commercial and open-source.

There are limitations with certificates (as with keys). In particular, we can’t enforce passphrase complexity (or indeed any passphrase at all!) and we can’t stop users from sharing keys. Just because a key has an identity user_sweh doesn’t mean it was me who used it; I could have shared it with Fred and John! Some of this is education (we teach people not to share passwords; we need to start teaching them not to share keys). Some of this can also be mitigated by shorter durations for human owned keys (if I need to get it re-signed every 90 days then it’s less likely that a shared cert will be usable).

But the ability to centrally control the options on a key as part of the signature (as opposed to relying on the correct settings in authorized_keys) is big, and allows for better auditing.

Special Thanks

I want to thank everyone who commented or emailed on the original post. I learned from them and they got me digging deeper into certificate capabilities. Thank you!

X-Forwarded-For and IP Allow-List

Wed, 13 Oct 2021 16:45:31 -0400

What is IP Allow-Listing

Typically when you want to access a remote resource (e.g. login to a server) you need to provide credentials. It might be a simple username/password, it could be via SSH keys, it could use Mutual TLS with client-side certificates… doesn’t really matter.

One concern is “what happens if the credential is stolen”. IP allow-listing is a way of restricting where you can use that credential from. “You must be coming from 66.228.55.57 in order to login as Stephen”.

This works quite well when restricting server-to-server communication, because servers tend to have fixed addresses (human-to-server wasn’t so useful because desktops or home ISP connection IP addresses may change frequently). Well… servers used to have static addresses, but in the modern cloudy world this isn’t always true any more. I jokingly call IP allow-listing a 20th Century solution because of this. However sometimes you still want to use it.

It’s simple, right?

In the early days, servers would communicate directly with each other, perhaps with a firewall in the middle

In this model your server can see exactly the IP address of the client machine (“Oh, you are 66.228.55.57; OK, I’ll let you login as Stephen”).

Nope!

Unfortunately this type of design is not really secure or performant. You may have load balancers in the path; you may (should!) have WAFs in the path; there may be proxy servers (on both the client and server side!).

Now your server only sees the IP address of the load balancer! So we need a solution for this.

X-Forwarded-For

This is a de-facto header for https communications that can be added to the request header. So in the above diagram the WAF would add a header

    X-Forwarded-For: 66.228.55.57

The load balancer would then add an extra entry for the WAFs IP address

    X-Forwarded-For: 66.228.55.57, 10.10.10.100

So now your server can look at this header and determine the original connection address.

So we can trust the left-most entry?

Unfortunately not. If the client had their own X-Forward-For header then this would appear first

    X-Forwarded-For: <client_values>, 66.228.55.57, 10.10.10.100

This might legitimately happen if the client was sitting behind a proxy server

    X-Forwarded-For: 192.168.100.42, 66.228.55.57, 10.10.10.100

Worse, an attacker could present their own values

    X-Forwarded-For: <attacker_provided>, 66.228.55.57, 10.10.10.100

And these fields may not be well-formatted resulting in a real mess.

    X-Forwarded-For: "ha ha I'm, attacking, you, 66.228.55.57, 10.10.10.100

So how do I use this header?

The first step is to not use it, unless you need it. If you’re in the first scenario (server to server, e.g. inside your company, or via trusted VPNs or fixed links to partner companies) where there is no WAF/LB in the path then you must not use the X-Forward-For header; it’s not trustworthy.

The next step is to start working from the right. So we start with knowing the load balancer is talking to us, and has presented the string

    X-Forwarded-For: 192.168.100.42, 66.228.55.57, 10.10.10.100

Hmm, we know “10.10.10.100” is the internal address of the WAF. So we ignore that. Hmm, “66.228.55.57”? Nope, that’s not one we know so we’ll use that. That’s the IP address we can use for allow-listing.

This process works for a longer chain of trusted services; you might have a kubernettes environment with Istio; the Istio gateway proxy may add its own X-Forward-For header.

    X-Forwarded-For: 192.168.100.42, 66.228.55.57, 10.10.10.100, 10.10.20.53

Now we see the Load Balancer address, but the same parsing process still works; start at the right and remove the addresses we know are ours.

Remember, this string may be badly formatted so ensure whatever you’re using to parse this line is robust!

Summary

IP Allow-listing may be considered a mitigating control for credential theft. You can’t always use it because of cloudy infrastructure and dynamic addresses. And intermediate infrastructure such as WAFs and load balancers make it harder to know the client IP address. The X-Forward-For header is a solution to pass this information on, but you have to be careful how you trust it!

Explaining technology as a story - TLS Certificates

Sun, 26 Sep 2021 16:11:30 -0400

When people ask me something technical, I frequently find it useful to tell the basics as a story or an analogy. Obviously all these stories have limitations to how accurate they can get, but it’s surprising how well it gets people to understand what you mean. So this post is part of a series of “explaining technology as a story”

TLS Certificates

One of the challenges, on the internet, is to know who you are talking to. If you’re about to enter your credit card details to Amazon then you want to be sure you’re talking to Amazon, and not someone pretending to be. Similarly if you want to login to your bank you need to be sure you’re really talking to your bank.

We also want to be sure no one is listening in; TLS handles both of this but I’m only concerned around the identity aspects.

This isn’t a new problem and different solutions have been used over the years. Letters would be sealed with molten wax and stamped with a signet ring. As long as the letter was still sealed and the recipient recognized the impression in the wax then there was a level of confidence that the letter was authenticate. Well… unless the ring was stolen, or had been duplicated, or some real clever person was able to open and reseal the letter with no evidence of tampering.

On the Internet, a TLS certificate does the same thing. Instead of a wax seal impressed with a signet ring we have a cryptographic signature.

What is identity?

Identity means different things to different people…or computers.

At one level it could be “I have gone to sweharris.org… am I looking at that site?“. This is the lowest level of identity. It doesn’t tell you anything about who is operating the web site. It’s really meant for the assurance that the site that came back was the site you requested. It won’t help you if you typed the wrong name, or clicked on a link that looked right but isn’t.

“Hi, I’m sweharris.org and here’s my signature that proves it”

If this part doesn’t match then the browser will throw up a nasty warning message to try and protect you.

At another layer it could help identity the person or company or runs the website. This is meant to be for the human using the web browser to be sure that when you go to a site you have a level of trust that you know who is running it

“Hi, I’m sweharris.org, which is run by Stephen Harris and here’s my signature that proves it”

The browser can’t help so much with the “organisation” part of this; that’s for the human to verify. It’s not clear this is a big win because of how it’s normally presented to the end-user. Indeed some browsers have stopped showing this “extended” identity at all.

Both of these identities can be stored in a certificate.

Certificate Authorities

Now there’s no way you can recognise all the millions of “signet rings” that would be needed. Especially when they change yearly! So, instead, we have a smaller number of people we trust, and we get them to sign it. So really what the web site sends is something closer to

“Hi, I’m sweharris.org and here’s Harry’s signature that proves it”.

We call these trusted signers “Certificate Authorities”, frequently abbreviated to CA.

Certificate Store

That all sounds well and good, but how do you know the CA is one that can be trusted?

This is one of the harder parts. We still have hundreds of “signet rings” that can be used to create signatures. Fortunately your operating system and browser handle most of this for you. e.g. in Chrome on Linux I can go to settings/Privacy and Security/Security/Manage Certificates (chrome://settings/certificates) and then click on the “Authorities” tab). There are over 70 organisations listed there, and many of those have multiple “Root” certificates.

So now it’s something like

“Hi, I’m sweharris.org and here’s Harry’s signature that proves it, and you can check it’s a trusted signature in your CA Store”

You can think of this CA store as a book of trusted people. And you can add your own CA Roots (it’s not uncommon for enterprises to have their own internal CAs). These will work the same as a public CA for trust purposes.

And you can even add a specific certificate, so now we have

“Hi, I’m YourServer.example.org and you can trust that because you know me”

Intermediate Certs

Of course it’s never that simple. For security reasons the CA Root certs are rarely used to directly sign end certificates. What really happens is that the CA creates an Intermediate certificate and this is what is used to sign the end certificate.

So now the conversation is really something like

“Hi, I’m sweharris.org and here’s Fred’s signature that proves it, and also here’s Fred’s proof of authenticity and that has been signed by Harry, and you can check Harry’s signature in your CA Store”.

This chain of trust can extend to multiple layers if necessary. It’s important that the server sends all these intermediates as a “chain” so the client can follow the chain to a trusted root.

That’s a lot of trust! Can we trust this is trustworthy?

This is where it gets a little bit messy. The main browser and CAs got together and formed the CAB Forum. This is a self-regulatory group that defined a set of rules that the CAs have to follow. If they don’t then the browsers will remove the Root CA and now the CA customers certs won’t be trusted.

Some of those rules are around how the CA verifies the customer is who they say they are. If I asked a CA for a signature claiming I was www.google.com then they should reject it because I can’t prove I own that domain.

Unfortunately with over 70 public Roots there’s a fair sized footprint for an attacker to try and get one of them to mis-sign a certificate, and some of the verification processes themselves depend on weak controls such as DNS (e.g. if I could compromise your DNS then I could create a certificate and request a signature from a CA that trusts DNS).

Since this is an “all or nothing” type scenario (“is there a chain of trust back to any Root in my CA Store?“) a compromise of one CA is pretty bad.

This is a recognized weakness of the current trust model, where ultimate trust is out of the end-user control (basically concentrated in the CABforum) and with weak processes. But it works pretty well and protects trillions of dollars per year in ecommerce globally. I think it’s good enough!

Summary

There’s a lot more to certs than just identity assertion (a lot of maths for one, along with public/private keys, renewal, multiple names, wildcards, client side certs…) but I’m mostly focusing on the story around verifying the identity of a server and how a typical browser does the job.

Explaining technology as a story - DHCP

Sun, 29 Aug 2021 08:38:21 -0400

DHCP

For a machine to be able to talk over IP it, naturally, needs an IP address. And that address needs to be unique on the local network.

Now for small networks you might be able to do this by manually configuring the address on each machine (e.g. on a home network you might do something like “my desktop is 192.168.0.10; the printer is 192.168.0.11; the laptop is 192.168.0.11; the X-box is…“). You might also want to do that if you have a bunch of servers where you need to know the address.

But this can be a pain, and it’s a lot of work to keep up. Even in a home environment do you really want to have to allocate everything? Your desktop, your laptop, your phone, your smart-TV, your DVR, your X-box, your IoT light-switch, IoT bulbs… I look at stuff in my house with an IP address and there’s a lot.

So what we do is use DHCP (“Dynamic Host Configuration Protocol”).

When a machine is added to the network (e.g. plugged into to an ethernet port, or connected to WiFi) the machine will send out a message, asking for what IP address to use (“Who am I, on this network?”).

The process has been given the name DORA, after the four steps:

DISCOVER - the client calls out to the local network “Hey, can anyone tell me what my address is?”
OFFER - A DHCP server on the network sends a reply “Oh, hi! If you like you can have address 192.168.0.100”
REQUEST - The client tries to confirm this “Hey, server, is it OK if I use address 192.168.0.100?”
ACKNOWLEDGE - The server responds. “Yep, that’s fine. Oh, you might also want to know these are the details default route, and the DNS servers, and the time server and so on. BTW, check back in no later than 3 hours later just in case this has changed”

If this fails for any reason (maybe the DHCP server isn’t working, or it doesn’t think there’s any free addresses) then the client will try a few times and then give up. At that point you might notice you get no address or you might get an address in the 169.254.. range (Windows, for example, will use the 169.254 range).

At roughly the half the check back time the client will try to refresh the values. It does that by jumping straight to the REQUEST part; “Hey, server, can I keep using this address?” If everything is OK the server will respond with its normal “That’s fine” response. If the server doesn’t like it then it might respond with a NAK (“No, that’s not good, ask for a new address”) and the client will go back to the DISCOVER part.

This refresh process is useful. It allows the server to update the client (eg if you changed your DNS servers, for example, the clients will eventually pick up the change). It also allows for the server to know if a client has left the network, so it can re-use the address elsewhere.

Static configuration and Dynamic DNS

Many DHCP servers, including the one built into many home routers, allows you to reserve addresses. For example, in an office you might have a printer on the same network as workstations. While the workstations can change address, the printer should always get the same one. To do this you would use the printer’s MAC address (this is the low level address built into the ethernet or WiFi card) and associate it with a fixed address. Now at the OFFER state the DHCP server will always respond with that address.

Now the REQUEST phase can be a little more complicated. When the client is requesting an address it can include additional information, such as “BTW, my name is ‘Stephens_Desktop’“. The server can use this information to update the DNS server’s configuration, so other machines can use the name. You don’t need to know what IP address a client received if the DNS is automatically updated.

Summary

In many offices desktop segments may have a Windows based DHCP and DNS server, which are also hooked into Active Directory. When a desktop is rebooted or a laptop connected it will use the DHCP server to get all the necessary details to talk on the network.

This process is fundamentally the same as when you turn on your X-Box at home, getting an address from your home router which typically has DHCP and DNS servers built into it.

Large or small deployments; DHCP is very common!

Explaining technology as a story - DNS

Sat, 21 Aug 2021 08:35:52 -0400

DNS

The internet basically runs on numbers (either IPv4 or IPv6). When your desktop connects to a server, or a web site, or pretty much anything then it uses the IP address. But humans prefer names; what is easier to remember, www.google.com or 2607:f8b0:4004:82f::2004? DNS is typically used to convert from the name to the number.

In a way you can think of this like a phone book. In the old days the phone company would ship you a large book (or two; “white pages” and “yellow pages”). If you wanted to phone someone and didn’t know the number then you’d look it up in the book and then dial the number. You might even have your own personal phone book with entries like “Mum and Dad”. With a modern smart phones you can search your address book for a name and the phone will be able to call them without you entering the number. This is what DNS does for the internet.

But the internet is massive; you can’t ship a “white pages” equivalent to every computer in the world. So it needs some clever lookups. So we need something similar to a telephone operator (“Goose Island, Oregon, please. The number for Dr Robert Humes”). We call these operators a “DNS Server” or “DNS Recursive resolver”.

Side note: In the early early days we did have a white pages equivalent called HOSTS.TXT. But eventually the scale of the internet made it unfeasible. It went away (much like the old phone books; I don’t think I’ve seen one for a decade!) but remnants survive in /etc/hosts or c:\windows\system32\drivers\etc\hosts.txt and similar places.

Now these DNS servers still don’t have a complete list of every address, so they have to ask other people.

Basically the server story goes like this:

I need to look up www.google.com. Hmm, I don’t know that.
Let’s see. Do I know who to ask about google.com? Nope!
OK, what about com? Can I ask someone there? I don’t no!
Hey, “DNS root”! I know about you… can you tell me who to ask about com?
<gets answer back; writes it in a book>
Hey, com server? Can you tell me where google.com information can be found?
<gets answer back; writes it in the book>
Hey, google.com server? Can you tell me about www.google.com?
<gets answer; also writes that in the book>
Great! I know the answer now.

This is quite clever. It means that each operator only needs to know about things one level below it in the tree. It doesn’t need to know about everything.

Now each time something is written in the book the server also writes down other information (e.g. when the entry was written, and how long to trust it for).

Next time someone asks the server for www.google.com it will look in the book and see “hey, I just found that out 2 minutes ago; I still trust it, and will return the same result”.

But what if someone then asks for maps.google.com? The process gets shortcutted…

I need to look up maps.google.com. Hmm, I don’t know that.
Let’s see. Do I know who to ask about google.com? Yes! I just looked that up a minute ago!
Hey, google.com server? Can you tell me about maps.google.com?
<gets answer; puts that in the book>

To make things more efficient, your local machine may also keep a small copy of the address book. So when you lookup up www.google.com it will remember it, and next time you want to go to www.google.com it will use its local copy and not even ask the operator. The web browser itself may even have an address book it keeps track of!

All these address books have to know how old the entry is and how long to trust it for. This is called the “Time To Live” (TTL). After an entry is older than the TTL then we should basically forget what we knew and ask for new data. This is how changes eventually make it across the internet so if a server moves to a new address eventually everyone should get the new data.

Summary

This story only reflects the barest basics of how DNS works (I haven’t gone into resource types, signing, delegation, glue records, and so on) and even this simplified version isn’t really accurate. There’s lots of edge cases in DNS; it’s really complicated!

But I find this story acts as a starting point.

Although I wonder how many people have even seen white/yellow pages outside of a movie? :-)

Explaining technology as a story - Routing

Sun, 15 Aug 2021 20:51:05 -0400

Routing

Far too frequently there are internet routing issues. Sometimes it gets bad that a large fraction of people can’t work (e.g. can’t see the corporate VPN gateways). In a recent occurrence some of the people I was chatting with didn’t understand what that really meant and why it impacted some users, but not others.

So I came up with a story about getting around town.

Imagine you don’t know how to get to where you want to go. All you really know is your local street, and how to get to the end of it. So what you do is go to that intersection, where you find a nice person. They ask you where you want to go, and then tells you to turn left, or to turn right, or to go straight on. You go down that street until you get to the next intersection and the same thing happens. Eventually you’ll get to your destination.

Now some of these people directing you are pretty dumb; they just say “I dunno, go that way and someone else will help you.” But others have a lot of information; they’ve got a big book of shortcuts and know a quick way.

These books aren’t fixed; they get updated all the time.

When there is a routing problem on the internet what has happened is that some of these kind people have had instructions crossed out in their book. Or worse, they may have received bad instructions and are sending you the wrong way.

Eventually the problem will be fixed and all these helpers will get their books up to date. But until then some people might be directed correctly, other people might be sent the wrong way, and yet others might just be told to give up; “you can’t get there from here!”

Summary

Obviously in this story we’re talking about packets and routers, and dynamic route updates (eg BGP). And I haven’t gone into any of the details.

But I think this gets across the basics of routing!

Summary of my current Home Automation Setup

Sun, 08 Aug 2021 18:52:08 -0400

I was asked to describe the stuff I use for my not-so-smart home and how it fits together. This was originally an email, but I figure other people might find it interesting

This is as complete as I can think!

The goal, where possible, is to have everything under local control and not dependent on the cloud. Setup may require cloud…

Lights

Philips hue bulbs where possible (including the mirror light)
- Hue hub has a local API
- Mostly Hue White bulbs (LWB014), a couple of Hue White Ambience (LTA003) and a mirror light (LWT018)
- Hue dimmer switches (RWL020) for normal “light switch” type control.
- Hue Motion Sensors (SML001) downstairs bathroom, laundry, attic
Lutron Caseta switches elsewhere (eg for outside garden lights, fluorescent bulbs)
- Really a cloudy service, but people have reverse engineered the protocol to allow for local communications, at least for monitoring and control. Requires cloud to add new switches
Smart switches (https://www.amazon.com/gp/product/B079FDTG7T/)
- flashed with Tasmota firmware for local control
- The original firmware is chinese cloudy but Tasmota firmware replaces this and provides local control via API or MQTT
- NOTE: Tasmota is not a supported firmware and newer revisions of the switch may not be flashable! Do research.
Ikea TRADFRI bulb 950lm
- This is a Zigbee bulb and can be added to the Hue Bridge. It’s a lot cheaper than Hue bulbs, but can’t be upgraded from the Hue Bridge.
- Pairing isn’t always easy (it took me many attempts) and it sometimes falls off the bridge and needs a power cycle.
- Doesn’t support the “on power failure/restore return the previous state” function of Hue Bulbs.
- I wouldn’t consider these for bedrooms because of that.

Plugs:

TP-Link HS105
- Really cloudy (Kasa) but people have reverse engineered the protocol for local control
Generic (https://www.amazon.com/gp/product/B07N1JPPXK/)
- Flashed with Tasmota. See previous warning!
Power sensing (https://www.amazon.com/gp/product/B07LGSBFNJ/)
- Flashed with Tasmota. See previous warning!
- I’m not sure these are accurate in what they report, but they’re good enough to let me know when the washing machine or drier has finished

Thermostat

Radio Thermostat CT50 with Wireless module
- I got this ‘cos it has full local control API… but it’s not very good; it seems like it only has a single-threaded CPU/OS so when it’s busy doing one thing then it can’t handle another, so API calls frequently get dropped. It works, but I don’t recommend it.

Garage Door

Shelly-1 with magnetic reed switch.
- https://www.sweharris.org/post/2019-05-19-garage/

Other

Frigidaire Air Conditioner FGRC0644U1
- This isn’t a bad aircon, but the mobile app isn’t the best. It’s totally cloudy with no local API.
- Has Alexa integration
Toshiba Microwave ML-EM34P
- As a microwave it’s OK. As a smart microwave it’s pretty bad.
- Totally cloudy
- Alexa integration frequently breaks (so much so, I stopped using it)
- The MSSmartLife app on mobile device isn’t even working properly!

Home Grown Stuff

Lots of things based around ESP8266/ESP32 with sketches I’ve written, to either report information or try to turn dumb devices semi-smart
eg

More complicated stuff:

https://github.com/sweharris/huebridge-shell
- It presents a “fake Hue Bridge” which is good enough for Alexa to discover. The lights it presents are handled by scripts. So if I say “Alexa turn bluray on” then it will turn on the fake light called “bluray” and my script will turn on the TV, turn on the BluRay player, turn on the Denon receiver and switch it to BluRay input. This is also how I can control the Tasmota switches by voice.
- I run 3 copies of this! One on my Mac for media player stuff, since it also can control itunes; one on the Home Automation (HA) VM for thermostat, aircon, tasmota devices, Garage Door; one on my desktop Linux machine so I can turn the screen off.
https://github.com/sweharris/Alexa-Smart-Home-VirtualButtons
- Using Lambda functions to present fake “door sensors” to Alexa which can be triggered via HTTP calls. When these sensors trip an Alexa routine can be triggered (eg say “Garage door is open”).
(unplublished) Media-API Alexa Skill
- “Alexa tell the robot …”
- This is written in GoLang and runs on my Mac Mini media player and is mostly for media control (“tell the robot to pause”). It’ll work out the current device (eg BluRay, or iTunes, or DVD Player App, etc) and send the appropriate commands. Also can be used for reporting (“ask the robot what for temperature”) where it will query MQTT for current values. This is kinda parallel to all the home automation stuff.

Putting it all together

A lot of this stuff communicates via MQTT, so I run Mosquitto MQTT server (https://mosquitto.org/) on the HA VM. Since I use CentOS 7 for the VM, I just use the version in EPEL

And then I run Home Assistant (https://www.home-assistant.io/) in a Docker container on the HA VM. This is what handles the coordination and triggers, as well as providing a dashboard. It also provides a mobile app so I can control things from out of home, as well as receive notifications (eg my phone plays a noise when the garage door opens). Talks to Hue, Caseta, TP-Link, MQTT directly without cloud requirements. Can talk to the Frigidaire cloud via a third party integration.

The Alexa Skill requires access from the internet so a firewall hole needs to be opened (in my case I had a webserver exposed, so I made it also act as a reverse proxy to send the request from external to internal).

Home Assistant requires access from the internet if you want to use it out of home, so needs a firewall hole as well. OR you can sign up for a cloud account and have them proxy the data. I didn’t want cloud dependency.

Examples:

“Alexa open the garage doors”
- Alexa -> HueBridge-Shell -> Shelly 1
Alexa announcing “Garage Door is open”
- Shelly 1 (reed switch) -> MQTT -> HA Automation -> VirtualButton -> Alexa Routine
Walk into downstairs bathroom, light turns on
- Motion Sensor -> Philips Hue Bridge -> Light
Turn on basement main light, second light over computers turn on
- Caseta -> HA Automation -> Caseta
- (this is how two unrelated switches can be put together to act as if they are one)
Walk into laundry, Hue Bulbs turn on, fluorescent bulbs turn on
- Motion Sensor -> Philips Hue Bridge -> Light -> HA Automation -> Lutron Caseta
- (an example how two different ecosystems can be put together)
Turn off Laundry lights with switch, Hue bulbs turn off
- Caseta -> HA Automation -> Philips Hue Bridge
- (another example of two different ecosystems)
“Alexa what’s the temperature”
- Alexa routine -> “ask the robot for temperature” -> Media-API
- (an example of how Alexa routines can now call skills)
“Alexa bedtime”
- Turns off the media center, nearly all the lights, my Linux machine monitor, the ground floor air-con; turns on the bedroom lights, starts the bedroom Echo playing music
- Alexa routine -> Philips Hue Brige + Caseta + HueBridge Shell (all 3 copies) + …
- (basically uses a lot of the stuff, all together in one routine!)

Conclusion

The current state of home automation involves a lot of patchwork and kludges. At least if you’re doing it on a budget!

I’ve got it working, and it’s pretty reliable. But it’s not totally hands free (I’d estimate maybe once a month I have to poke at HA; less so once I got rid of the Alexa integration and wrote my own VirtualButton skill).

What I’ve done works for me. And somehow the HA VM has gone from being a test (“how does this work”) to part of my daily home experience. It does stuff in the background that I only realise if it breaks!

Making a doorbell semi-smart

Thu, 08 Jul 2021 20:04:31 -0400

If you’re anything like me then you sometimes miss the doorbell ringing.

It may be because you’re engrossed in a movie and the doorbell sound doesn’t register. Or you might be (in these COVID times) in a make shift home-office with the door closed and an air-con blasting out. Or you may even be out of home.

Can we add some home-automation smarts to a dumb doorbell?

How does a doorbell work?

Your typical doorbell is powered by a transformer; maybe 16V AC. Yes, it’s an AC circuit. This causes many complications for smart devices, and even lighted doorbell switches (which need a diode to cause the light to work!)

Side note: A smart doorbell like a Ring needs to draw enough current to power the smart electronics, but not enough to cause the doorbell to actually ring. This is sometimes an art, given the sheer variety of doorbell chimes, so it’s not uncommon to hear complaints of “buzzing”.

This is my transformer:

But the doorbell circuit itself is pretty simple. It’s a a simple loop between the transformer and the chime with a button in the middle. When you press the button the circuit is complete and the chime rings!

Now a number of houses have two doorbells. They’re effectively two independent circuits. They share a common line to the transformer but have different connections to the door chime.

And at the door chime itself we can see how the three wires all get together:

Measuring this

Now if this was a simple DC circuit then we could (with suitable resistor dividers) just measure the voltage on the wire going from the doorbell to the chime. But it isn’t; it’s AC. So we’re going to have to be a little more clever.

Fortunately there exist sensors that can measure current. There are some that sit in the circuit, and there are some that can wrap around a cable and use a “hall effect” sensor to detect current flow.

This latter is what I used. The one I used looks a lot like this and uses a “CT103C” sensor:

This provides a small analogue signal that can be measured by a simple sketch on an Arduino or similar. Because I have two doorbells and wanted WiFi this meant I needed an ESP32 (the ESP8266 I normally use only has one ADC channel).

The idea is that you put the wire from the doorbell to the chime through the center hole and reconnect it to the chime. Don’t put it on the common wire to the transformer ‘cos then you won’t know what button was pressed.

In practice I found there wasn’t enough space inside the chime cover to hold both sensors, so I used a simple extension wire to let it go outside of the chime case.

You can now do an analogRead() on the pin. Typically when the button isn’t pressed you’ll see a value close to zero (it may be a little higher if you have a lighted doorbell button). But when you press the button the value jumps a lot. In my case it went to over 400. This makes it easy to detect a button push.

Other considerations

We also want to add a short “debounce” delay; this will stop “unclean” press/release from triggering the event too quickly. And, hey, we’ve all seen that impatient person ringing the doorbell multiple times per second… we can stop that as well!

Because of my home automation setup, I just made the ESP32 send a message to my MQTT server. The exact topic that will be used will depend on the MAC address of the ESP32 used, but in general it will send a “pressed” message

The resulting sketch is here.

What can we use this for?

In Home Assistant we can set up automation tasks. In the automation section, a setup such as

- alias: Back Doorbell
  initial_state: true
  trigger:
    - platform: mqtt
      topic: doorbell/123456/back_doorbell
      payload: 'pressed'
  action:
    ....

- alias: Front Doorbell
  initial_state: true
  trigger:
    - platform: mqtt
      topic: doorbell/123456/front_doorbell
      payload: 'pressed'
  action:
    ....

is sufficient. We don’t need to create sensors, just have the automation triggered directly.

Now we can do anything that Home Assistant is configured for.

If you use my Alexa Virtual Buttons skill then you can have Alexa trigger anything it controls (even speak “Back door bell pressed”). Or you could have flash lights (useful for deaf people). Or if you have a smart TV then you could pop up a message; e.g. with an LG OLED TV and my code fork you can do something like:

  action:
    - service: rest_command.alexa
      data:
        command: "pushcontactbyname"
        param1: "Front Door"
        param2: 0
    - service: shell_command.tv_toast
      data:
        message: "Front doorbell"

as the action. If you’re out of home you could have it send a text message, or use the Home Assistant application to create a notification (although they’re not so quick, sometimes!)

Logging

Home Assistant also logs activity. In the logbook you may see entries such as

Back Doorbell has been triggered by mqtt topic doorbell/123456/back_doorbell
6:59:19 PM -

Or if you’re into SQL…

MariaDB [hass]> select time_fired,event_data from events where event_type='automation_triggered' and event_data like '%doorbell%mqtt topic%';
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| time_fired                 | event_data                                                                                                                  |
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------+
| 2021-07-08 22:59:19.544307 | {"name": "Back Doorbell", "entity_id": "automation.back_doorbell", "source": "mqtt topic doorbell/47C708/back_doorbell"}    |
| 2021-07-08 23:00:08.090737 | {"name": "Front Doorbell", "entity_id": "automation.front_doorbell", "source": "mqtt topic doorbell/47C708/front_doorbell"} |
+----------------------------+-----------------------------------------------------------------------------------------------------------------------------+

You now have a record of every time your doorbell was pressed!

Summary

This isn’t a fully smart doorbell… but it gets part way there, and isn’t dependent on any cloud provider. Local control!

Digital Safe version 2

Tue, 18 May 2021 16:12:56 -0400

Previously I modified a cheap electric safe to work with an Arduino, because the original controller board had failed.

But because my build skills aren’t the best, I kept getting serial port issues, and more than once needed to get to the emergency key to open the safe.

At the same time someone asked me if the same design would work with one of the real cheap ($30) safes on Amazon.

So I wondered if I could do better. This time using an ESP8266 board, which provides WiFi access. No serial ports, and the board can host all the software. A quick test of putting one inside the safe (powered by a battery pack) showed only a small attenuation in the WiFi signal; it was still strong enough to talk to the network.

I had also been asked if some of the cheaper safes might be convertable.

Requirements

A cheap Safe
An ESP-12E
RFP30N06LE MOSFET
A 4.7kOhm resistor
A N4004 Diode
A board to solder it to
Some wires and connectors

The MOSFET, diode and resistor aren’t size sensitive. I just picked these ones ‘cos they felt right. The important part is the MOSFET needs to trigger at 2V and be of N-channel type (to switch the -ve side of the circuit).

The Safe

This is the safe I got. It’s about $30 from Amazon. The model I bought was ‘cos it was $2 cheaper, but there’s a lot of identical models on Amazon.

This is the inside of the safe. The red button on the left is what you would press to reset the combination. But we’re interested in the two screw holes. A standard phillips head will let you remove these two screws.

You also need to remove this screw. At this point the cover can be removed. It’s a bit tight because of the rubber plug that’s there to stop the door rattling, but a bit of brute force will get it out.

And this is what the internals look like. The magic is the blue solenoid. In the “down” state it lets the locking gate slide sideways. And that’s how this type of safe works; the key (as shown in this picture) pushes down on the solenoid. The electronic part activates the solenoid and that lets the gate slide.

This is what it looks like in the locked position.

The only cable we care about is the two wired to the solenoid.

Disconnect those from the board. FWIW the other 4 wires handle the power (red/black) and the reset button (white). We don’t care about these, though; we only want the two wires to the solenoid. So carefully unplug that cable. That’s it!

At this point we could remove the circuit board, but let’s leave it in place. We can always revert the changes and switch the safe back to normal operation just by plugging the cable back in again.

NOTE: This solenoid is especially easy to “bounce” by banging on the safe; the spring is very weak. That means that the right timing of banging on the top and turning the handle might open the safe. This isn’t high security! (The key lock also isn’t high quality). Increasing the security of the safe isn’t part of this instruction!

Annoyingly it’s not clear what the solenoid is. The blue wrapper blocks all data. So how much power does it need? We know it can work on 6V (‘cos of the 4 batteries) but how much current does it need? I measured the resistance (15ohms). At 6V that’d be around 400mA which made me wonder if the Vin line of the NodeMCU would be sufficient. And… it wasn’t. It was enough to hold the lock open but it wasn’t enough to cause the solenoid to activate. So we’re gonna need to be more innovative about powering the system.

Designing the replacement board

So let’s look at designing the replacement.

For power, let’s use a MOSFET as a relay. I used a RFP30N06LE because this can be triggered at 2V and so the 3.3V output from the NodeMCU should work.

To make sure we have the order of the pins on the MOSFET right, looking at the front (with the writing facing you and the pins downwards) the 3 pins are:

    Gate   Drain   Source

Source will be connected to ground, drain will connect to the solenoid, and gate will connect to the NodeMCU pin we use to trigger this. The other pin of the solenoid will connect to the power. Now when the gate line goes high the solenoid will trigger.

Finally there’s a 4k7 resistor between gate and ground. This is a pull-down resistor just in case the ESP-12E pin floats. It’s possibly not needed, because the software drives the pin high or low and if the software crashes then the watchdog timer will reboot it pretty quickly, but I put it there just in case.

That all sounds complicated! Now to work out the best placement for a veroboard so I can solder it all together. Which appears to be:

That doesn’t look too complicated. One diode, one resistor, one MOSFET, a few cables, and some pins.

Prototype

So I prototyped this on protoboard. This is powered by a 9V battery, which won’t be the final solution! I removed the solenoid from the safe to make it easier to test.

(Note: there’s no diode here; I added it later! And if you’re looking carefully the resistor is in the wrong place. Apparently I got lucky and didn’t destroy the MOSFET… Also shush spot the cable from D7 when the diagram says D6. You didn’t see that! How this prototype ever worked… I dunno!)

In this setup I can do

    curl http://safe.local/safe/?open=1

and the solenoid will open for 5 seconds. And… it worked!

In this case I used a NodeMCU ESP-12E even though it’s overkill and oversized. You could use a WeMos D1 Mini or clone and it would work just a well. However the D1 Mini will only take 4->6V input, whereas the NodeMCU ESP-12E can take up to 10V (maybe even 12V). That gives us more flexibility in the power supply we use. I did my testing with a 9V battery, which wouldn’t have been possible with the D1 Mini.

Note that D6 on the ESP-12E “conveniently” lines up with the Gate on the MOSFET. Not like I planned that or anything, honest! (Confession I had planned on D7 being the pin but I counted wrong and D6 it is. Which is why the proof of concept used the wrong pin. Oops!).

Now I wanted to use a “socket” like approach for the ESP-12E so I could remove it for software upgrades. This just required 3 pins (GND, Vin, D6) but I used 4 (the second GND) to add a bit of stability. This turned out to be one of the harder parts because I couldn’t find any single-pin sockets that are big enough to handle the standard pins. I ended up cutting up a 40pin strip, and that’s not very tidy… but it worked.

(In the second of these boards that I made I used a couple of 8 pin strips and cut the pins off and covered the gaps with insulation tape; that also worked and was easier to handle).

I also put a couple of normal pins on the board to make it easy to connect to the solenoid via normal male->female connector cables, and I soldered relatively thick power cables that can be connected to the power supply.

(and yes the resistor is still in the wrong place! but at least the diode is there, now)

And this is how it looks with the solenoid connected and the ESP-12E inserted.

My soldering skills still aren’t the best… but, at least it works!

Mounting

Unfortunately there’s not enough space inside the safe cover on this small safe (the old large safe had plenty of free space) so now we get to the mounting part.

To run the solenoid cables from inside the safe cover to outside it I cut a small hole near the top of the battery door. Just big enough to slide the two wires through. This is the only real change to the safe.

I connected two “dupont compatible” male-female cables to this connector.

Mounting of the whole thing was… done by duct tape. Yeah not very professional but it seems to work.

Importantly I taped down the power wire so that it didn’t flex whenever the door was opened/closed. This means the wires shouldn’t snap off the board. Whenever movement happens it can stress solder joints, so we want to prevent that as much as possible. Let the tape take the strain!

And the ESP-12E just slots in, nicely. It’s easy to remove if software updates are needed. Just be careful to align the pins correctly when you plug it in.

Now the power cables can feed out of the back of the case. I was worried that the relatively thin metal could rub and end up damaging the cables so I protected them with some more tape. And I taped the wire to the top edge of the box. I didn’t do a very good job, really. Not pretty… but it works!

And then out of the back I added more tape to try and keep it in place.

Powering this safe can be done pretty much from any DC power supply from 5V to 10V (maybe up to 12V). I had a really old Blackberry power supply (I never had a Blackberry, so not sure where I got it? Maybe from a “junk” bin at work). This is 5V 700mA. Plenty. I cut the tip off and connected the wires up.

It’s very important to ensure you get the +ve side connected to the +ve input to the board, because the ESP-12E may not like reverse voltages. This power supply didn’t make it obvious which wire was +ve so I had to use a multi-meter. But once I knew which was which it was simple to connect.

And to protect this also from movement I stuck it to the back. Lots of duct tape used in this build!

And that’s it! The safe is built.

Software

Now we come to the software

The safe is modeled with some simple states:

Unlocked (no password set; safe can be opened any number of times)
Unlocked once (safe can be opened once, then goes back to being locked)
Locked (password is needed).

The solenoid lock is always kept in the locked position unless the “open” function is selected. Open can not be selected if the safe is Locked.

This will let you go to http://192.168.4.1 or (if you’re lucky) http://safe.local and that will display the WiFi configuration screen.

Enter your WiFi SSID and password. In 5 seconds the software will reboot and the safe will connect to your local network. It will attempt to get an address by DHCP.

If it is unable to connect to the network after 60 seconds then it will give up and start up the Safe-123456 access point again and allow you to change the network details and try again.

Opening the web page in a browser should give you a menu on the left and a larger status box on the right.

The first thing you should do is go to the “Change Safe Authentication Details” link at the bottom of the menu. Here you can set a username and password for your safe. It’s recommended you do this so that only authorised people on your network can reach this interface. After setting the password you should be prompted to login before doing any more work.

On the main menu you will see various options:

Status” will tell you if the safe is unlocked, locked, or one-time unlocked.
Open safe” lets you open the solenoid so the door can be opened. This will not work if the safe is locked.
Unlock pswd” has 3 options. If you get the password wrong for any of these then there is a 2 second delay before the safe will respond again.
- “Test” - this will let you test the password but leave the safe locked. This may be useful to use after setting a password so you can be sure you can unlock it again, but before closing the door.
- “Unlock Once” - this will let you select the “Open Safe” function once. After that the safe will go back to locked mode and you’ll need to re-enter the password to unlock it again. This is the normal function to emulate the original safe.
- “Unlock Permanent” - this will remove the password totally and let you use the “Open Safe” function as many times as you like. This is similar to pressing the “red button” inside the safe that lets you change the combination.
Finally the “Lock Safe” function lets you set a new password for the safe. This can only be done if the safe is unlocked. If the safe is locked or “unlock once” then it will be rejected.

Note I’ve been saying “Password” and not combination. That’s because we’re not limited to numbers and we’re not limited to 8 digits.

Passwords can be up to 100 characters long. Although I’ve tested with a few special characters (eg. % and &) you should verify the password with the “Test password” function before closing the safe door!

Talking to the safe programmatically

Although the software wasn’t designed to provide an API, all the access is via HTTP requests and so can be performed from the command line (eg curl).

The format is

    http://safe.local/safe/?command=1&amp;parameters...

The /? are important.

Of course you need to provide the username and password.

So, for example, to get the status:

    curl -u user:pass http://safe.local/safe/?status=1
    Safe is unlocked

Commands you might use:

    ?open=1&duration=<time_in_seconds>
    ?lock=1&lock1=<pswd>&lock2=<pswd>
    ?pwtest=1&unlock=<pswd>
    ?unlock_1=1&unlock=<pswd>
    ?unlock_all=1&unlock=<pswd>

Note that “lock” needs the password sent twice ‘cos it checks the two match.

Now you might think “if I can send unlock_all requests in a program then I can write a loop to test them all”.

Yes… but, good luck! An 8 digit numeric password has 100,000,000 combinations. Let’s assume you get it right half way through. 50,000,000 attempts. Each attempt takes 2 seconds ‘cos the safe code enforces that. That’s around 3 years. And the password can be digits and letters and more. You might get lucky! But don’t bet on it…

Conclusion

And that’s it!

These cheap safes can be converted in pretty much the same way as the larger one. I’m not sure what use these safes are, though, because they’re not very secure.

I think I’m gonna replace the Arduino/Relay board in my other safe with another one of these ESP8266 based boards, though. Should be a lot more reliable than the old design, has less cables and isn’t chained to the computer by a serial port!

Update with a different board

It’s been pointed out to me that ESP8266/Relay boards exist.

On these boards the relay is typically connected to D4. You’ll obviously need a USB->Serial (3.3V!) programmer to program this type of board since they don’t come with their own serial interface. But the sketches I provide should work with it with just the pin change.

This is probably a easier option; no soldering necessary. A little bit more hassle to program the device, but a lot cleaner!

Using RSA and ECDSA on Apache with CentOS / RedHat

Sat, 08 May 2021 17:25:57 -0400

Previously I described a relatively modern set of TLS settings that would give an A+ score on SSLtest. This was based purely on an RSA certificate.

There exist another type of certificate, based on Elliptical Curve cryptography. You may see this referenced as ECC or, for web sites, ECDSA. An ECDSA certificate is smaller than an RSA cert (eg a 256bit ECDSA cert is roughly the equivalent of a 3072bit RSA one). This makes it quicker to send, although at present RSA signatures are quicker to verify than ECDSA.

My setup

I use LetsEncrypt for my certificates, although (for complicated reasons) I don’t use their standard certbot client. Instead I use Dehydrated. I like this client because it just puts the certs into a directory and then you can do whatever you like with them (in my case I use ansible to distribute to lots of different endpoints).

The Dehydrated client also supports ECDSA certificates (indeed, in the current version it defaults to secp384r1 to generate 384bit certs).

Configuring Dehydrated for both cert types

So in my case I configured the app to default to rsa and then created an override.

   % cat domains.txt
   *.sweharris.org sweharris.org > sweharris
   *.sweharris.org sweharris.org > sweharris_ecdsa

   % grep KEY_ALGO config 
   KEY_ALGO=rsa

   % cat certs/sweharris_ecdsa/config 
   KEY_ALGO="secp384r1"

With this setup when I run the “refresh” command (dehydrated -c) it will place the RSA cert in the sweharris directory and ECDSA cert in the sweharris_ecdsa directory:

   % openssl x509 -noout -text -in certs/sweharris/cert.pem | grep Public.Key.Algo
               Public Key Algorithm: rsaEncryption

   % openssl x509 -noout -text -in certs/sweharris_ecdsa/cert.pem | grep Public.Key.Algo
               Public Key Algorithm: id-ecPublicKey

Inside the certs directory we will find four files of interest. These are always symbolic links to the latest version, so that provides a consistent name for us to use:

cert.pem
This is the public key for the server certificate
privkey.pem
This is the private key for the server.
chain.pem
This file contains the “chaining” certificates needed to generate a chain of trust to the root.
fullchain.pem
This file contains the chaining certificates and the server public key. Sometimes this can be easier to use.

We’ll see, next, how these files are used.

Configuring Apache

I have two primary server types in my environment. They’re either based on CentOS 7 or CentOS 8. The configuration is slightly different.

The certificates

For CentOS 8 it’s pretty simple, we can use the full chain file:

   SSLCertificateFile    /etc/httpd/conf/sweharris/fullchain.pem
   SSLCertificateKeyFile /etc/httpd/conf/sweharris/privkey.pem

   SSLCertificateFile    /etc/httpd/conf/sweharris_ecdsa/fullchain.pem
   SSLCertificateKeyFile /etc/httpd/conf/sweharris_ecdsa/privkey.pem

Now if you try that on CentOS 7 it seems to work, but SSLtest complains about missing chain certificates and small DH key sizes. Not sure why that happens. But fortunately we can use a slightly different path

   SSLCertificateFile /etc/httpd/conf/sweharris/cert.pem
   SSLCertificateKeyFile /etc/httpd/conf/sweharris/privkey.pem

   SSLCertificateFile /etc/httpd/conf/sweharris_ecdsa/cert.pem
   SSLCertificateKeyFile /etc/httpd/conf/sweharris_ecdsa/privkey.pem

   SSLCertificateChainFile /etc/httpd/conf/sweharris/chain.pem

This works because both certs are signed by LetsEncrypt via the same chain (same intermediate, same root). If that wasn’t the case then we’d need to work a little harder to ensure the chain file had everything that was necessary.

The ciphers

The previous cipher list I provided only selected RSA options; after all, why bother with ECDSA options if you don’t have the right certificate? But now we do. As with the RSA certificates there’s a small number that can be considered “strong”. Indeed, we’ll stick with two

   ECDHE-ECDSA-AES256-GCM-SHA384
   ECDHE-ECDSA-AES128-GCM-SHA256

There are others which maybe considered (e.g. CHACHA-POLY based), but I don’t believe the oldish versions of openssl on CentOS currently support them.

This results in a cipher suite configuration of:

   SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256

You may notice that i no longer have DHE-RSA-AES256-GCM-SHA384 listed as a cipher, but I still have a couple of CBC based ones there.

The CBC ciphers are still needed because, as we previously saw, older versions of Safari need them, and these old versions don’t work with ECDSA

   Safari 6 / iOS 6.0.1
   Safari 7 / iOS 7.1
   Safari 7 / OS X 10.9
   Safari 8 / iOS 8.4
   Safari 8 / OS X 10.10

The result

The good news is that we still get an A+ rating! The report also shows the two options for the certificates.

It correctly reports the 6 ciphers as being available

   TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c) ECDH secp256r1 (eq. 3072 bits RSA)  FS	  256
   TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)   ECDH secp256r1 (eq. 3072 bits RSA)  FS	  256
   TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b) ECDH secp256r1 (eq. 3072 bits RSA)  FS	  128
   TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)   ECDH secp256r1 (eq. 3072 bits RSA)  FS 	  128
   TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)   ECDH secp256r1 (eq. 3072 bits RSA)  FS  WEAK  256
   TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)   ECDH secp256r1 (eq. 3072 bits RSA)  FS  WEAK  128

Of course the two CBC ciphers are still flagged as weak, but we’re aware of this and they’re needed for Safari compatibility.

Digging further into the report I noticed that most of the clients negotiated ECDSA mode. Only a handful did RSA mode!

  Chrome 49 / XP SP3	TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  Safari 6 / iOS 6.0.1	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
  Safari 7 / iOS 7.1 	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
  Safari 7 / OS X 10.9 	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
  Safari 8 / iOS 8.4  	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
  Safari 8 / OS X 10.10	TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384

Literally everything else that SSLtest emulates that can work with with TLS1.2 all negotiated ECDSA. Interestingly “IE 11 / Win Phone 8.1” previously needed a CBC cipher but it can do ECDSA in GCM mode!

Complication

Although this doesn’t impact me, in an enterprise environment the support for multiple certificate types may be impacted by middleware boxes such as load balancers that do TLS termination (so they can inspect the traffic and do sticky sessions or similar). It’s not much point in doing ECDSA if other devices can’t support it! Similarly you may need the DHE based ciphers to handle limitations of those devices. But the list provided here should be a good starting point.

Conclusion

This didn’t take too long to figure out, once I stopped fighting the CentOS 7 build and attempting to use the fullchain.pem file there.

It’s now possible, even for the relatively old CentOS 7, to support modern ciphers and modern public key cryptography. Every cipher used supports forward secrecy and large key sizes. We just need to be careful about any future CBC Oracle attacks!

Of course ECDSA isn’t resilient against quantum computing attacks, but I really don’t think that’s going to be a practical concern for a long time.

Note the full report includes a secondary domain (*.spuddy.org) so doesn’t quite match what I’ve written. I didn’t include a description of that because it’s unnecessary complication. In this case a client without SNI (are there any that also support TLS1.2?) would get a cert for a different virtual host and so get an error. That’s also not relevant to this post :-)

Data Loss Prevention (DLP)

Thu, 06 May 2021 09:11:41 -0400

Working in Cyber Security I’m frequently reminded that the reason we do all the things we do is, ultimately, to protect the data. After all, apps are there to process data, servers (and clouds) are there to run apps and store data. So the whole of cyber security is there to protect the data. It may be Identity and Access Management (restrict access to data to those people who should have access to it). It may be anti-virus or endpoint malware detection (prevent ransomware from destroying data). It may be encryption (prevent attackers from bypassing other controls and reading data directly). We have secure software development processes to prevent attacker getting access to data via application bugs. We have WAFs to detect and block common patterns (eg SQL injection). And so on.

Traditional DLP

Each of these “pillars” of cyber security are critical and all form part of a DLP posture. But in this blog post I’m going to focus on the area most traditionally associated with the moniker “DLP”; preventing data “theft” and how the cloud makes this a lot more challenging.

In this realm we’ve traditionally focused on protecting the data at rest and in preventing unauthorized data from leaving the company (the challenge, of course, being to permit authorized data transfers; don’t block the business from working!).

Our threat actors are staff making mistakes (“oops; I sent this confidential email to the wrong address”), malicious insiders (e.g. a salesman planning on getting a new job trying to take confidential documents with him) through attackers who have a footprint in your environment and want to use that to exfiltrate your data.

Border

A standard way to assist in DLP is to create a strong border around your assets. There’s firewalls between you and the internet, between you and any partners you may have connections to, potentially even between parts of your company (why should HR staff have connectivity to trading systems?), and definitely between desktops and the internet. Of course you need internet access so you create bridges across the border between the internal and external networks, such as web proxies. Traffic is funneled through these proxies and can be inspected. Bad traffic can be blocked.

The challenge, here, is that these controls are relatively easily bypassed.

Those that work on traffic inspection (“Hmm, this looks like a bunch of credit card numbers; we shouldn’t let that out”) can be defeated by a simple obfuscation (map 1 to !; map 2 to " and so on). Encrypted files are yet another challenge for traffic inspection. There are some clever tricks that can increase coverage (require encrypted files to use a managed solution with keys the DLP agent can access) but in general they’re a “best efforts” type solution. They’ll definitely help in catching mistakes but a competent malicious person can bypass them. The stricter you make the rules the harder you make it for the company to do business.

Similarly controls that work on volumetric patterns (“Huh, this machine is consistently sending 10Mbit/s of traffic for the past day”) are defeated by slow exfiltration techniques.

And to make things even worse, not all traffic can be proxied so the firewall, itself, may need to inspect traffic. TLS1.3 (and good TLS1.2 settings) is designed for Perfect Forward Secrecy (PFS) and so the firewall can not just decrypt the traffic as it flows through.

Where is your border?

But all this presumes you know where your border is. Do you? The cloud changes all this. If you spin up an AWS VPC and create an Internet Gateway to that VPC then you’ve added a new border to your environment. Every vNet in Azure automatically has a gateway and so a border. If you use an S3 bucket then that’s potentially exposed to the internet; a border! Other cloud native services? Borders!

Even if you think you know all your borders then the posture may change without you knowing. In March 2021 Amazon introduced EC2 Serial consoles which let you access the console of your EC2 instance. This is really useful if you have broken your network config; it lets you get in and fix things. The problem is that this serial connection can be exposed to the internet via SSH. It requires a malicious person to do this (since the config needs to be refreshed every minute to persist the gateway) but this connection is direct to your VM bypassing all your firewalls, your processes. It’s a new border! Oops! (At time of writing there’s no way to disable this SSH gateway without blocking serial console access totally because of how AWS has implemented the technology; there’s an enhancement request in!)

Other borders may be hidden. If you use Lambda functions (pretty common!) then additional triggers may be added. In particular, HTTP API gateways provide a way for the function to be called from the internet. Again it bypasses your firewalls and WAFs; it’s a new border. Amusingly, Alexa triggers can be added to Lambda functions. An intruder could potentially persist a foothold in your environment as an Alexa call (“Alexa, open the backdoor to megacorp”; “Alexa read out all the stolen credit card numbers”). Are you monitoring for Alexa triggers attached to your lambda functions? You should be; it’s a border that needs to be controlled!

So we can see that controlling the border is a lot more complicated in the world of cloud. All of these new borders need new technologies to detect, block and control. Your traditional firewall/web-proxy solution won’t work.

Zero Trust

You may look at Zero Trust Networking as the solution to this; if every endpoint is doing the inspection and validation of traffic then do we need a border? In my view the answer is “yes”. For example, developers make mistakes. Also not all technologies can be fully controlled (e.g. BMC ports, out of band consoles) to the same level as a VM (which can have agents installed).

We also have a much bigger configuration management problem to deal with, including how to be sure you’ve got 100% coverage (which in turns requires an accurate inventory of all assets, which is a hard problem!), activity logging and so on.

We’ve moved from needing to control a small number of border crossing points to controlling tens of thousands of endpoints; every server, every database, memcached, key/value stores… a green-field environment may be able to create an architecture that would work, but for the rest of us in brownfield waste dumps… nope! A lot of enterprise technology of the past 2 decades was designed with the assumption that it would be protected from the outside world (single factor LDAP authn? Ugh) and can’t easily be fixed.

Configuration management

Configuration management also applies to cloud environments. So many cloud providers allow changes to be performed from web consoles. Indeed many changes are easier from the console because they call dozens of backend APIs to get the job done. Where possible you want to deploy and manage your cloud environments programmatically (e.g. using Terraform) but even there you must validate the deployed configuration hasn’t drifted because of a mistake by an admin. This is critical for environments such as “Microsoft 365”; even if you use Powershell to manage the environment it’s possible for an admin on the console to accidentally click the wrong button. Monitor your configurations and alert if they change!

Encryption

But what about encryption? Can this help with DLP? Yes… and no!

As I wrote in Data At Rest Encryption encryption can be deployed at different layers.

Block Level. This is useful for laptops (“oh no, I left my laptop in the taxi”) or mobile devices where the risk of physical loss of a device is something to be mitigated, but it doesn’t really help in a datacenter environment where the loss of a disk isn’t that high a risk.
Database level. This can help against an attacker getting access to the server, but isn’t a strong defense against someone with database credentials
Application level. This now requires the application itself to be compromised (so you need credentials and encryption keys to get the data) but has some downsides (e.g. full text searching may not be possible) so isn’t a good solution.

Remember that everywhere you store your data you need to protect it; that includes backups! Depending on how the backup is done and the encryption used, you may need to ensure your backups are fully protected. There has been more than one data leak from badly protected backups.

Cloud services complicate the encryption world even more. Many services have “server side encryption” options; e.g. S3, RDS, EBS in Amazon can all be encrypted. The problem with this encryption is that it’s closely aligned to “block level” or “database level” encryption; if you get valid credentials then the data is transparently decrypted. The Capital One breach in 2019 had data stored encrypted this way but because the attacker was able to get credentials they were able to get access to the unencrypted text.

I, personally, don’t consider cloud server side encryption to be sufficient protection for data at rest. Wherever possible use application level encryption.

Scan on upload

You should also try and scan data when it’s been placed in storage; e.g. putting data in an S3 bucket or in OneDrive/Sharepoint can raise an event that can trigger a scan of the object. If it’s been determined to be “bad” then remedial action (“delete!!”) can be taken programmatically. Of course this scanning still has the same problems previously discussed, but it will (again) detect mistakes.

Data Destruction

That’s not to say that cloud level encryption is useless. If you can encrypt it with a customer managed key (CMK) that you can delete then you have the ability to delete data (by cryptoshredding) when you no longer need it (e.g. exiting a cloud provider). Remember that data needs to be protected wherever it is; when you leave an environment you should ensure all traces are destroyed, and deleting the encryption key so any data remnants that may remain are unreadable.

This CMK requirement can rule out some services. For example some part of the Azure DevOps suite use shared services at the backend that are controlled and managed by Microsoft and so use Microsoft managed keys. Your source code may be confidential data and a corporate asset; do you want to trust it to an environment where you can’t be sure of deletion?

SaaS and DLP

And then we get to SaaS offerings. These providers need access to your data so they can process it. If you just sent them encrypted blobs then they won’t be able to do much work. So, remembering that data needs to be protected everywhere, you need to look at how the SaaS provider protects it. Do they let you use a CMK? Are you comfortable with their encryption processes? How is access to the SaaS offering managed? This is effectively a new border! How can you control this one!

Conclusion

As we can see, DLP is not an easy topic. There’s no good answer here, just various levels of “not so good”.

And this is where risk management comes in. What is the impact to the company if this set of data is exposed (including reputational impact, which still applies even if the data was encrypted) versus the benefit to the company of processing data in this way. Because of the limited border crossings in the traditional world we had a pretty good idea of the size of the risk and we knew how to mitigate many of them (e.g. endpoint security controls and agents; remember all the cyber pillars are involved!).

But in this new world where data is everywhere and every location has unique controls and traditional solutions can’t be deployed, we need to look at alternate mitigations and controls and evaluate the risk on a case by case basis.

DLP is hard!

More modern TLS settings

Wed, 23 Sep 2020 19:30:16 -0400

Back in 2016 I documented how to get an A+ TLS score.

With minor changes this still works.

But times have changed. In particular older versions of TLS aren’t good; at a very least you must support nothing less than TLS1.2.

Consequences of limiting to TLS 1.2 or better

If you set your server to deny anything less than TLS 1.2 then sites like SSLlab tell us that older clients can no longer connect.

In particular:

    Android 2.3.7
    Android 4.0.4, 4.1.1, 4.2.2, 4.3
    Baidu Jan 2015
    IE 6 / XP
    IE 7 / Vista
    IE 8 / XP
    IE 8-10 / Win 7
    IE 10 / Win Phone 8.0
    Java 6u45, 7u25
    OpenSSL 0.9.8y
    Safari 5.1.9 / OS X 10.6.8
    Safari 6.0.4 / OS X 10.8.4

Now these are very old. We know that Win7 is still popular, so should we worry? Not so much; IE 11 on Win7 can support TLS1.2.

So I think it’s reasonable to disable anything lower.

Stronger ciphers

We also see that SSLlabs flag CBC ciphers as weak. Now this is debatable. CBC isn’t necessarily weak. But it’s historically had a lot of bad implementations, leading to variations of POODLE attacks (amongst others). It’s kinda expected that there will be yet another variation of POODLE on CBC. So SSLlabs flag these ciphers as weak, as an encouragement to move people off and onto stronger ciphers.

Unfortunately, it turns out there aren’t that many! At least not if you’re using RSA certs (which most people do).

In Apache/OpenSSL terms these ciphers are pretty much the only good ones:

     ECDHE-RSA-AES128-GCM-SHA256
     ECDHE-RSA-AES256-GCM-SHA384
     DHE-RSA-AES128-GCM-SHA256
     DHE-RSA-AES256-GCM-SHA384

Yes, that’s it. 4 of them.

If you run this through SSLlabs then it’ll be strong, but a number of clients won’t be able to connect.

    IE 11 / Win Phone 8.1
    Safari 6 / iOS 6.0.1
    Safari 7 / iOS 7.1
    Safari 7 / OS X 10.9
    Safari 8 / iOS 8.4
    Safari 8 / OS X 10.10

(Notice IE11/Win7 isn’t on that list; it’ll happily do secure ciphers).

You need to decide if you want to support these older clients. If you do then these two CBC ciphers will make everything work:

    ECDHE-RSA-AES256-SHA384
    ECDHE-RSA-AES128-SHA256

Yes, these get flagged as weak, but now every client that supports TLS1.2 should work.

Complications

In an enterprise environment the client rarely talks directly to the server, especially not over the internet. There may be load balancers or WAFs in the path and these may be the ones that terminate the TLS connection. So their configuration is important.

In particular, F5’s BigIP servers can only do DH key exchange on 1024 bit keys. Which is considered weak. 2048bit is the minimum. Their rationale is that 2048bit keys require 5 times the computation than 1024bit keys. Which makes me wonder if the “big” in “BigIP” is now valid!

This is a problem. We have to choose between weak key sizes or potentially weak ciphers. In this scenario I recommend enabling CBC mode and ensuring you’re always on top of all patches; if a new POODLE variant is discovered, patch ASAP!

Conclusion

For my CentOS 7 Apache servers I now have the following configuration:

    SSLProtocol TLSv1.2
    SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256

SSLlabs tells me this results in:

    TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)   ECDH secp256r1 (eq. 3072 bits RSA)   FS            128
    TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)   ECDH secp256r1 (eq. 3072 bits RSA)   FS            256
    TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x9e)       DH 4096 bits                         FS            128
    TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x9f)       DH 4096 bits                         FS            256
    TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)   ECDH secp256r1 (eq. 3072 bits RSA)   FS   WEAK     256
    TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)   ECDH secp256r1 (eq. 3072 bits RSA)   FS   WEAK     128

All the ciphers have forward secrecy, all are at least 128bit in strength, and all the clients that support TLS1.2 will connect.

Note the DH key is 4096bits in size. Why can’t BigIP support that?!

It was OK before; why is it broke now?

Sun, 09 Aug 2020 12:07:30 -0400

As I was rebuilding my network I came across a problem.

In my basement I had previous run a cable from my core switch around the room to the other side, where I had a small 100baseT switch to handle the equipment on that table. I’d also run another cable across the ceiling to the back of the house, where I had the Powerline network.

Everything seemed to be working fine, and it had been doing so for years.

But when I upgraded the equipment to handle gigabit I was annoyed to see both of these links only connected at 100Mbit speeds. WTF? I checked the cable itself. That’s fine; it’s cat5e. I did a pin test. That’s fine; all 8 pins had proper continuity, no breaks.

So then I looked closer. WTF? I had wired the pairs wrong.

A normal UTP cable (defined as T568) has the pin pairs as 12 36 45 78. Somehow I had wired them 12 34 56 78. So this meant the original RX pair were not across a single twisted pair, but across two separate wires.

At 100Mbit speeds this didn’t seem to matter. But at Gigabit speeds it just didn’t work. Cutting off the bad ends and crimping new ones on solved the problem.

Vendor Risk Management

The next day, at work, I had to have a similar discussion, but around vendor risk.

Normally when we look at a vendor (say, we’re outsourcing data processing, or selecting a SaaS provider) we look at the type of data we will be using with that vendor, and look at their controls and determine if they’re good enough. After all, we don’t need as much control around a site that just has pictures of the CEO’s cat doing funny things as we would around a processor of credit card information.

We’ll document what we found out about the vendor, the risk management team will do their due diligence and give an approval.

Then a year later someone decides “Hey, these guys are cheap, let’s move all our HR ID photos into their system”.

At this point alarms should start ringing.

A new evaluation of the vendor is now required. You’ve changed the scope of the engagement, the riskiness of the data they’ll be processing. What was OK for the cat pictures may not be OK for HR data.

It’s not just data; it could also be access. A site that was accessed by two people… we could get away with local admin control. But if we then open up that site to a thousand people (“let’s allow all staff members to upload cat pics”), we’re going to need to look at access admin and controls. We might even need an approvals workflow, to prevent staff from uploading inappropriate photos.

So quantity can also impact control requirements.

But not all changes do; if the CEO got a dog and wanted to make it “cat and dog pictures” we haven’t appreciably changed the scope, and a new risk evaluation wouldn’t be needed.

Summary

Don’t assume that what was good in the past will be good after a change has been made. In my case the cable that worked fine for years was no longer suitable.

Similarly, engagement changes may require a re-evaluation of the current service to determine if it’s good enough.

Extending the wireless on my router

Sat, 25 Jul 2020 18:34:00 -0400

3 years ago I replaced OpenWRT with a home grown router.

It’s worked pretty well, but I wanted to take advantage of improvements in networking (5Ghz!) and also improve coverage. This kinda became important due to COVID lockdown and Work From Home. My library, where I was working from, had very weak network signal. I needed to do better.

So I decided to look at turning off in the inbuilt WiFi and use an external WAP (Wireless Access Point, or just AP). However I wanted to keep using the Multi-SSID setup I had, to allow for LAN and Guestnet access (and since that post I also added an IoT network, although I’m not sure I’m gonna keep it).

The obvious way of doing this, on Linux, is to use 802.1q VLANs and then have the AP map the SSID to the VLAN. In that way traffic from the LAN network would still appear on the br-lan bridge on the router and everything would work the same way.

I might write another post, later, explaining 802.1q but for now you can imagine it’s a way of having multiple virtual LANs all running over the same cable. Each virtual LAN gets a number between 1 and 4095, and the switches can decide what ports get to see the traffic. This allows you to create a backbone network, but still only pass relevant traffic to different network segements. What I’m doing, in this post, is defining a network backbone of 3 VLANS, but endpoints (eg my desktop, or a phone on the guest wifi network) acts is if it was on a dedicated network.

Switches

Now this requires switching equipment to also be able to understand 802.1q tagging and be able to make some ports dedicated to various networks, and make other ports “trunk” networks, able to pass traffic for multiple networks.

I picked the TP-Link SG116E switch for this. It’s a 16 port Gigabit switch and can handle 802.1q VLANs. I got two of these; one for my basement (where the existing switch is) and one to replace the living room switch.

Now, arbitrarily, I chose VLAN 10 to be my main LAN, VLAN 11 to be for Guestnet and VLAN 12 to be for IoT.

Router config

On the Linux router this is done by configuring “sub-interfaces”

% cat ifcfg-enp3s0.10 
DEVICE=enp3s0.10
TYPE=Ethernet
ONBOOT=yes
BRIDGE="br-lan"
NM_CONTROLLED="no"
VLAN=yes

This simple configuration will add the “enp3s0” sub-interface 10 to the br-lan and any traffic sent out of it will have an 802.1q tag value of 10.

Interfaces “.11” and “.12” are similarly configured.

So now the bridging table would look something like

bridge name     bridge id               STP enabled     interfaces
br-guest        8000.000db9439cce       no              enp3s0.11
br-iot          8000.000db9439cce       no              enp3s0.12
br-lan          8000.000db9439ccc       no              enp3s0.10
br-wan          8000.000db9439ccd       no              enp2s0

(I’ve hidden the hostapd entries for WiFi since I’m gonna turn that off in a little while, and it’ll just confuse things)

The wisdom of building these bridges, earlier, is now paying dividends. I can add new networking constructs to the bridges and it “just works”.

Back to the switch

I now need to configure the switch to understand this. I plug the router port 3 (“enp3s0”) to the switch port 1. And I configured the switch so that port 1 is tagged VLANs 10, 11, 12 and every other port is untagged VLAN 10.

At this point we haven’t gained anything; everything on the wired network is still all LAN traffic. But we’ve got VLAN tagging between the router and the switch. So we’ve now got potential.

Next step is to do similar for the switch in the living room, behind the TV. I similarly configure that switch so port 1 is tagged 10,11,12 and the rest are untagged 10. I can now configure basement port 7 (don’t ask!) to be tagged VLANs as well.

So now all the VLANs exist in both the main switches, even though all the client machines are on VLAN10 (LAN) only.

Access Points.

I’m now ready to start putting APs on the network.

I looked at mesh networks, but most of the kit wasn’t able to do multi-SSID to VLAN mappings. They’re great at coverage, but if I can’t do the networking constructs I want then they’re useless to me. It’s possible the Ubiquiti products can do this, but they’re very expensive. If you want prosumer, go that path. I decided to go down the DIY path.

Which meant delving back into the OpenWRT world. Some research (and a sale at Amazon) lead to buying 3 TP-Link Archer A7 routers. These can very easily be flashed with OpenWRT. They can then be configured to act as APs and not routers. It took a few trial and errors to understand what the document and examples meant when it came to multiple VLANs and multiple SSIDs, but eventually I got it.

I avoided re-using constructs the base software uses (eg the WAN port, or the network names WAN and LAN, and VLANs 1 and 2). So I’m just going to use the 4 LAN ports and build three networks Main, Guest, IoT

Only the Main network needs to have an address ‘cos that’s how you’ll talk to the AP later.

Defining AP VLAN ports

OpenWRT being Linux, you have similar sub-interface addressing for VLANS.

We also need to tell the switch what ports are on what networks.

BE VERY CAREFUL HERE. If you get the setup wrong you can lose network connectivity to your AP and then have to fight to get it back. I’m going to describe the target state but not how to get there in each step. It’s worth leaving at least one port on LAN and keeping that configured so the original 192.168.1.1 address is still visible on that port… just in case :-)

But at the end of the day, in the main main network/switch menu you will end up with a config similar to:

This tells the AP to configure all 4 LAN ports as tagged for 10, 11, 12. In this case I have port 4 plugged into the main switch, which is also set for tagged VLANS. VLAN1 was the original VLAN used by the software to manage the LAN, and VLAN2 is for the WAN. It’s possible (but I haven’t tested) to remove VLAN2 from the WAN port and add that to 10,11,12.

The equivalent in /etc/config/switch is

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option vid '1'
        option ports '0t'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option vid '2'
        option ports '0t 1'

config switch_vlan
        option device 'switch0'
        option vlan '3'
        option vid '10'
        option ports '0t 2t 3t 4t 5t'

config switch_vlan
        option device 'switch0'
        option vlan '4'
        option vid '11'
        option ports '0t 2t 3t 4t 5t'

config switch_vlan
        option device 'switch0'
        option vlan '5'
        option vid '12'
        option ports '0t 2t 3t 4t 5t'

The numbers “0 1 2 3 4 5” on the option ports line refer to ports seen in the diagram, but the ordering can change on different equipment. On the A7, 0 is CPU, 1 is WAN, 2->5 are the LAN ports. This is common, but not always true! That’s why it’s worth doing that bit in the GUI, to let it pick the port numbers.

We can then define the 3 bridges (“interfaces” in OpenWRT speak) and associate them with the VLANs (also in the same config file).

config interface 'Main'
        option ifname 'eth0.10'
        option type 'bridge'
        option delegate '0'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '10.0.0.249'

config interface 'Guest'
        option proto 'none'
        option type 'bridge'
        option delegate '0'
        option ifname 'eth0'

config interface 'Iot'
        option ifname 'eth0.12'
        option proto 'none'
        option type 'bridge'
        option delegate '0'

So now we have the AP configured on the ethernet site and can be plugged into a trunk port on the switch.

Defining AP WiFi

Onto WiFi!

From the Network/Wireless menu you can simply add new networks to each Radio (2.4Ghz and 5GHz). The setup would look similar to:

Remember to set a WPA2 key on the security TAB. For me that meant adding entries for the Lan/Guest/IoT networks and selecting the relevant interface. It’s boring, but works.

At the end, in /etc/config/wireless you’ll see entries similar to:

config wifi-iface 'wifinet1'
        option device 'radio1'
        option mode 'ap'
        option network 'Main'
        option encryption 'psk2'
        option key 'MYPASSWORD'
        option ssid 'MYSSID'

You should see entries for each of the SSIDs and for both radio1 and radio0.

You can see the OpenWRT eventually built network structures very similar to what my original router config had:

# brctl show
bridge name     bridge id               STP enabled     interfaces
br-Guest        7fff.74da88a9a43b       no              eth0.11
                                                        wlan1-1
                                                        wlan0-1
br-Main         7fff.74da88a9a43b       no              eth0.10
                                                        wlan1
                                                        wlan0
br-Iot          7fff.74da88a9a43b       no              eth0.12
                                                        wlan1-2
                                                        wlan0-2
br-lan          7fff.74da88a9a43b       no              eth0.1

After all of this hard work, we’re kinda back where we started.

We have an AP presenting the same 3 wireless networks as before and they’re on the same router bridges as before, and so the same firewall and routing rules work.

Replication

So why go to all this hard work?

Well, now we can replicate this. I can put one of these A7s behind the TV in my living room (pretty much central to the house). I can put one in the laundry area of the basement at the back of my house. And I can put a third one in my library, with an ethernet cable running to the living room TV switch. I chose different frequencies for each one, so they don’t fight for airwave bandwidth!

Which is what I did. And it works… adequately. OpenWRT doesn’t currently have “beam steering” available, so when I walk from one room to another then my phone might decide to fall back to the slower 2.4Ghz, rather than stay on 5Ghz. And, oddly, machines don’t always connect to the closest AP (why would a smart plug in the laundry, literally 4ft away from the AP, decide to connect to the one in the library, 2 floors away?)

But despite this, it works well. I have strong signal everywhere.

Garage

With one caveat. My garage. When the metal door is closed it seems to reduce the signal strength just enough to cause my smart dumb opener to lose network signal.

So for that I used a TP-Link AC750 Powerline adapter. I found a plug in the basement (conveniently close to the basement A7) that was on the same circuit as the garage electrics. So I was able to use one port of the A7 to be untagged Guestnet and plugged the Powerline adapter into that. It works.

The power of 802.1q aware switches (which my core switches and the A7s are) is that they can extend and expose any of the networks, so I could even expose a hardwired Guestnet port from any of these switches.

Port Summary

Basement:

 VLAN  VLAN Name      Member Ports   Tagged Ports   Untagged Ports
 1     Default_VLAN 
 10    lan            1-16           1-3,7          4-6,8-16
 11    guest          1-3,7          1-3,7
 12    iot            1-3,7          1-3,7

Port 1 goes to the router; 2 is my “test” port and is currently unused, 3 connects to the basement A7 and 7 connects to the Living Room switch. All other ports are on the main LAN

Living Room:

 VLAN  VLAN Name      Member Ports   Tagged Ports   Untagged Ports
 1     Default_VLAN
 10    lan            1-16           1-2,12          3-11,13-16
 11    guest          1-2,12         1-2,12            
 12    iot            1-2,12         1-2,12

1 goes to the basement switch, 2 goes to the living room A7, 12 goes to the library A7. All other ports are on the main LAN.

Finally

And just for completeness, I have a dumb Gigabit switch in the basement on the far side from the router and core switch. This is used for my “exercise entertainment” center; I have an old old TV, an old Mac Mini, a TiVo Mini, a Pioneer Amp and a FireTV stick there. These need networks, so I ran a cable around the room and put a dumb switch (my original gigabit switch before I went VLAN aware) there.

So my final network looks like:

Firewall Basics

Sun, 01 Mar 2020 19:10:30 -0500

What is a firewall?

Think of an office building with a keycard entry system. To get into the building you need to put your card to the reader. If it think you’re authorized then the gate will open and you can enter. Similarly, a number of offices require you to “badge out” as well. This badge/gate system is a simple analogy for a firewall.

A network firewall may be considered to be a security gate to separate the network up into “inside” and “outside” and you define rules to allow traffic to pass through the gate.

The outside of a firewall is generally considered the less “trusted” environment. The inside is the more trusted. An obvious example is a firewall between your internal network and the internet. The internet is “outside” and the internal network is “inside”.

Traffic that flows from the outside to the inside is called ingress traffic. Traffic flowing in the opposite direction is called egress.

We also call the traffic that flows through a firewall to be “North-South” traffic.

Typically, in an enterprise, we control ingress traffic to restrict what an outside attacker can see. Similarly we control egress traffic to restrict what can be reached. Why would your database server even need to connect to the bitcoin network?

Internal firewalls on a network

Above I mentioned an obvious firewall location; between the internal network and the internet. However there’s also a benefit to splitting up your internal network.

A common design is the “3 tier network”.

The firewall between the internet and Tier 1 is commonly called a “border firewall”. The others are “internal firewalls”.

This doesn’t look like much. What’s the advantage of all these internal firewalls?

The advantage comes in when you have multiple applications on your network (which is common to enterprises). The webserver for “application A” sitting in tier 1 (DMZ) can have permission to only talk to the application servers in tier 2 (App) that run application A. In that way if the web server is compromised it can’t even see the app servers for “application B”.

Similarly if an application server is breached (e.g. an attacker manages to jump from the web to the app tier) then the only databases they can see are those that are used by the application. Application B’s databases can’t be reached.

Now this sort of design has some limitations. “East-West” traffic (traffic between servers sitting in the same tier) isn’t restricted. This leads to a more fine grained designed, sometimes called “micro-segmentation”, but that’s beyond the scope of this blog entry!

Types of Firewall

Firewalls typically operate at 2 layers of the network stack.

IP layer

The first operates at the IP layer, and looks at packets. When a client wants to make a connection it (typically) generates a local port and then does the equivalent of “I am 10.20.30.40 port 12345 and I want to connect to 192.168.100.200 port 443”. The firewall can see this and decide whether to allow the traffic through or not.

Things get slightly more complicated because a “session” consists of multiple packets and not all of them say “I want to connect”; some of them say “and I’m part of the conversation I’m already having”. A large packet may also be split up; if you send 64K worth of data then it’s normally split up into 1500 byte or smaller fragments and then reassembled.

This leads to three types of IP firewalls:

Packet filtering

This is the original design of a firewall. Basically it would have rules that say “ingress traffic to my webserver on port 443 is allowed; the webserver can talk out”. It’s how I did Cisco ACLs back in the 90s. It works to protect against ingress traffic against unapproved services (“no, you can’t reach my ssh port!“) but some protocols may be open to abuse from an attacker sending malformed packets.

Stateful packet inspection (SPI)

This is probably the most common form of IP firewall. Now the firewall keeps track of traffic previously approved, and allows related traffic to pass through.

In my home grown router config my iptables rules had:

iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate INVALID -j DROP

These use a “connection tracker” (conntrack) to keep track of traffic and only permit relevant traffic through. Now my web server doesn’t need to have a specific “allow to talk to the internet” rule; it can magically respond to incoming requests. This is clearly stronger.

We also drop traffic that claims it’s related to a session, but isn’t (eg attackers sending fragments, or trying to hijack sessions with the wrong values).

Deep packet inspection (DPI)

Now we start to look inside the traffic itself and reject stuff that looks bad. At the IP layer, itself, we can block malicious traffic from hitting the web server. Both filtering and SPI are done on the headers; DPI (as the name suggests) goes deep into the packet contents. It will collect together fragments and try to understand the content of a message. Does it look bad? “Why is there HTTP traffic on the SMTP port?” Block!

In a world that is trending towards “encrypt everything”, DPI is becoming less useful. But there’s still enough unencrypted traffic that it’s not without value.

Application layer

The most common use of firewalls at the application layer are the Web Application Firewall (WAF). These are protocol specific (in this case, http traffic) and know a lot about the types of attack that can happen at that layer (“SQL injection, buffer overflow, invalid headers” for example) and can block and reject the traffic before it even hits the webserver. An enterprise with thousands of webservers may take time to patch against the latest Apache Struts vulnerability (not to pick a product at random)… but your WAF can be programmed to detect a potential attack against Struts and block it. (But still patch your servers! Don’t rely on a single defense!)

WAFs are expensive to run, both in terms of compute and in terms of operation because they are prone to false positives, but they are a critical component in a firewall posture.

For the rest of this blog I’m going to focus more on the network layer because that’s where most people run into trouble.

IPv6

At this point I should point out that IPv6 is a different protocol to IPv4. Your network firewall rules written for IPv4 will not automatically protect against IPv6 unless you have a management stack that handles both.

This has some pro’s and con’s.

For my personal home network the IPv6 rules were pretty simple; “allow access from trusted IPv6 servers; reject everything else”. But the IPv4 rules had dozens of entries to handle all the ingress rules I permitted.

For an enterprise they need to be aware that migrating from an IPv4 only environment to a (normally dual-stacked) IPv6 environment will require ensuring their firewalls also cover the new stack.

What direction?

This is where I see a lot of developers get confused. They look at traffic as to “where the data flows”. For a lot of developers “I pull data from https://api.example.com" is considered an ingress. And from a dataflow perspective it is. But from a network perspective what matters is “who initiates the connection”. If my server is making the connection then it’s an egress rule that needs to be created; “my server 10.20.30.40 needs an egress rule to talk to 192.168.100.200 on port 443”

When creating network firewall rules you need to think at layer 4 of the OSI stack (TCP, normally) and not at layer 7 (the application layer).

Is a NAT router a firewall?

No. But kinda?

NAT is nasty. It breaks a lot of rules. But, in some respects, it provides a useful ingress firewall. That’s because the machines behind the NAT gateway can’t directly be reached from the outside; they’re not routable. No one on the internet can reach 192.168.1.100. But if I add “port forward” on the router so that 443 -> 192.168.1.100:443 now the web server on that machine can be reached.

So this has some of the benefits of an ingress firewall. Nothing can be reached unless you specifically permit it! So the SPI rules kinda apply. It’s not really a firewall but kinda acts as one.

Note that has home networks start to move to IPv6 then NAT may (should!) disappear. So some of the protections home networks have had due to NAT may disappear.

Is a proxy server a firewall?

Kinda, yes?

An enterprise typically uses a proxy server to permit egress traffic to “trusted” sources. The proxy can do content inspection and reject malware and traffic to known bad hosts. In some respects we can think of the proxy as the “other side” to a WAF; WAF protects ingress traffic, a proxy protects against egress.

In a strongly controlled enterprise, firewalls should block all egress traffic by default. Only specifically opened firewall rules should be allowed. And proxies form a strong part of this. There could be a set of rules for desktops, and each server could be specifically permissioned to only see the https endpoints it’s meant to see.

Proxies have a great advantage over network layer firewalls because they work at the URL level. A web service hosted in AWS could have a gazillion of IP address and you don’t want to allow your network firewall to open all those addresses ‘cos other AWS customers also use them. But the URL? That’s easier.

I consider egress proxy rules to be far superior in these days of modern elastic compute, and so proxies form a critical component in allowing the enterprise to consume external resources.

Is a reverse proxy server a firewall?

No.

A reverse proxy, typically, just takes incoming requests from the internet and directs them to a second server. It may look at the request and forward the request to different servers; it may load balance requests. But, in essence, a reverse proxy is just a “layer 7 router”. It doesn’t add any real security and, indeed, can reduce security by exposing backend servers that wouldn’t otherwise be reachable (because the firewall has blocked it).

There are some good functional reasons to have reverse proxies in your environment (e.g. to allow the creation of unreachable networks, and have this bridge the gap) but this doesn’t replace a firewall. Indeed it’s almost an anti-firewall!

Conclusion

A firewall segregates network traffic. It may operate at different layers of the network stack. It’s a critical component for any network security model.

Understanding the underlying network layers (e.g. TCP, UDP) and the application layer is important to building a strong firewall posture.

If you’re in an enterprise and need to create a “firewall request” so your server can be reached from the internet (or reach out to the internet!) then understand the layer 4 semantics; who is initiating the connection matters more than “who sends all the data”.

RSA wrapped AES

Sun, 15 Sep 2019 12:20:40 -0400

Here’s a common requirement:

We want to transfer a file containing sensitive data to a partner; they want us to put the data in their S3 bucket. How can we do this securely?

Now you might start with putting controls around the S3 bucket itself; make sure it’s properly locked down, audit logs and so on. But there’s a number of issues with this. In particular, S3 bucket permissions are easy to get wrong. There’s a long history of data leaks because of misconfigurations. Even if the setup is correct today, it may be wrong tomorrow. Since the bucket isn’t owned by “me”, I can’t verify the setup. So I’m transferring data into an untrusted environment.

So use encryption

Since I’m responsible for the data until it reaches the partner, I look at encrypting it, instead. Now even if the bucket is open by mistake, the contents can’t be read.

The obvious method of encryption would be to use AES. This has a downside, though. It’s symmetric encryption and so the sender and receiver need to have a ‘shared secret’ (the encryption key). This can be done, but it introduces complexity.

So let’s look at asymmetric encryption; RSA. This just means the sender needs to know the recipients public key. So far so good. But RSA is quite slow and CPU intensive. When sending lots of data this can have a performance impact.

Hybrid encryption

We can steal a technique used in various places.

The underlying idea is that the data is AES256 encrypted using a unique one-time-use key and then the AES key is encrypted by a recipient generated RSA public key. The encrypted data and the encrypted AES key are placed in the bucket. The recipient can then use their private key to decrypt the AES key, and then use that to decrypt the data.

The steps generally are:

The recipient creates an RSA key pair. This key can last a long time (e.g. 1 year)
The sender generates a random AES256 key. This key is only used once.
The data is encrypted with the AES key
The AES key is encrypted with the RSA public key
The encrypted data and the encrypted key is sent to the recipient (e.g. via S3)
The recipient uses their RSA private key to decrypt the AES key
The recipient uses the AES key to decrypt the data
The recipient deletes the encrypted data and encrypted AES key.

The benefits of this process:

No shared secret is required. The sender needs only know the public key of the recipient
Each data object is encrypted with a different key
The large data objects are encrypted decrypted with the fast AES algorithm; only the key, itself, uses the slower RSA algorithm.
If permissions are set incorrectly (eg on the S3 bucket) then the data is still fully protected even if an attacker obtains the data.

If the same data needs to be sent to multiple recipients then this process is easily extended; for each recipient use their public key and store multiple copies of the encrypted AES key, one copy for each recipient.

Requirements

The RSA key must be at least 2048bits in size (giving 112bits of strength); recommended to be 4096 bits (giving 128bits of strength).
If a “Key Derivation Function” (KDF) is used to generate the key then the password used to generate that key must be at least 45 characters long (assuming a character set of 64 characters - 6 bits of entropy).
If a “ZIP” tool is used to perform the encryption then it must support AES256 mode (e.g. SmartCrypt). Generic “zip” encryption is weak and must not be used.
We must be sure the public key used belongs to the recipient; if we use a key supplied by an attacker then the attacker will be able to decrypt the data

Typically the recipient can publish the key in a trusted location (e.g. on a TLS protected website under their control). This has advantages that the recipient can replace their key and we can retrieve the new version automatically.

The security of the solution is dependent on using the correct public key

Example using Unix native tools

In Unix, the openssl command does most of the hard work for us:

Generate RSA key:

Recipient generates the RSA key and extract the public key. This version has no password on the private key, so protection of the private key is essential. This only needs to be done once a year or two, on the recipient schedule.

$ openssl genrsa -out private.pem 4096
Generating RSA private key, 4096 bit long modulus
...++
............................................................++
e is 65537 (0x10001)
$ openssl rsa -in private.pem -outform PEM -pubout -out public.pem
writing RSA key
$ cat public.pem
-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAtIvkiOFgQfGYa2SIZCPO
dMg8qju6BRgpBAf7el8nr1D/0dG03CusYUzbd9j6/F07CriZpfO5XQE/0Bza8dyf
abZhXtxe8Sa9SUxaOzfyrXlBe5zvO+DSbQ2L326Nw6ScptsNM0yeDaMgfxp0/Kcx
3fW4U6s1RJ6D7B/XHJ6FZiIX4e4AKEoOWnJnNCKET9TsYn64TaL0zop3su5QzSbL
B70mrIJhme+9EVf91CdIU+efa4KF+bMiM8Kqgpb7UauI+5mWJN66JJ82i6xMz2hl
OOIysBZRJCG7UoURblQqgYMtBEhpX9DPZmAvfOWkqO7kouSLYT5S4ZRnys47Fi6K
Sj64XtMB63/HMCq0HXb6UXxm98mICA+vtVGVhJLt9nq7bmABaa+jluI27k62/hUI
WmpREiSt1U80gmU2c17PhQknVl2iioSpnO7p12U/xWq9oFJbnQ3OKgL1jxzQsfVD
U/wAnYiZiLm8Oro1yMd832LBBC01OcMG1v93FZ1g5UUiCecYhqTERRkkhYEltIML
01JDVxEJeWHrtzUdWNKV2pFBGedWuB0/hpJsub+Xcp4sycY4P8mSdFgL99uL1nJc
HsXDEZTS+DoRSo6x+Nf6ytSl7ShESERrubmlvbWrgUQTb/vzJIhwr/Ov1DvOjanU
kjBXt/d7eYhmeMgrHeB6itcCAwEAAQ==
-----END PUBLIC KEY-----

The encryption process:

Generate a random 32byte (256bit) key in hex format:

$ aes_key=$(openssl rand -hex 32)

Encrypt our data file with with this key. For aes256, GCM mode is preferred. However, my copy of openssl doesn’t support that, so we’re using a CBC mode. These are OK. Do not use EFB!

$ openssl enc -aes-256-cbc -K $aes_key -iv 0000000000000000 -in original_data -out encrypted_data

In this case we can use a fixed Initialization Vector (IV) of all zero’s because the AES encryption key itself is only to be used once. If we were using the key multiple times then we would need a random IV

Encrypt the AES key with the public key

$ echo "$aes_key" | openssl rsautl -encrypt -pubin -inkey public.pem -out aeskey.enc

The files aeskey.enc and encrypted_data can now be sent to the recipient (e.g. placing them in their S3 bucket)

To decrypt the data

The recipient now needs to decode the data, first by extracting the AES key. Note it uses the private key to do this:

$ aes_key=$(openssl rsautl -decrypt -inkey private.pem -in aeskey.enc)

Now they can decrypt the data file:

$ openssl enc -d -aes-256-cbc -K $aes_key -iv 0000000000000000 -in encrypted_data -out results

The file results matches the original_data

Conclusion

This wrapping of a random AES key within an RSA key is a pretty common technique for transmitting data.

In this scenario we’re not providing any checksum or validation that the data has been sent by us; we’re just ensuring that the data contents are kept confidential. This scheme could easily be extended by providing a checksum file signed with the sender’s private key. The recipient could use the senders public key to verify the checksum was properly signed, and the checksum matches the unencrypted datafile’s content.

However, the recipient may be content in knowing that only the permitted sender is allowed to write to the S3 bucket, and consider this additional step unnecessary. After all, they can see the bucket configuration!

Capital One Breach

Wed, 21 Aug 2019 11:47:37 -0400

I was asked a question around the Capital One breach. It seems that, in some areas, fingers are being pointed at Amazon, and they should be held (at least partly) to blame for this.

It also seems as if Senator Wyden is also asking Amazon questions around this.

There’s also a question around Paige Thompson, the hacker, and her previous relationship as an Amazon employee. If she used any insider knowledge to break into Capital One then this would erode a lot of trust in Amazon’s Web Services, and the public cloud in general.

And, finally, were code repo’s used in the attack? Is there a whole supply chain issue, here?

Now I’m not convinced about any of this. Pretty much all the chatter I’ve heard in the InfoSec communities was “It’s all Capital One’s fault”.

It’s not 100% clear from the indictment, but it looks like the attacker made use of SSRF on the WAF to access the ec2 metadata URL, which included role credentials and those credentials were overly broad and gave access to S3 buckets.

It’s also not clear if this was a AWS WAF, or a hosted WAF (eg Imperva WAF, F5 WAF) on an EC2 instance.

So the first set of questions I have are:

Why didn’t the WAF block the metadata URL?
Why was the WAF EC2 instance associated with an IAM role?
Why was the role overly broad?
Why wasn’t the data encrypted in the bucket?

Out of all of this, Amazon themselves may be partially implicated

If it was an AWS WAF, why doesn’t that block the metadata URL by default?
Did the attacker make use of knowledge obtained when she was working for Amazon 3 years earlier?

So it’s really really looking like a pure Capital One misconfiguration.

It is Interesting that Capital One created Cloud Custodian to try and detect misconfigs :-) It’s clear that detection is a hard topic, and not necessarily so good for those “unknown” edge cases.

Note: S3 server-side encryption is inadequate because the service will decrypt data automatically if you get the right credentials. I consider it the equivalent of “block device encryption” or “TDE - Total Database Encryption”; an SA or a DBA can still get to the data, and app level encryption is still required. Indeed, I wrote about this 2 years ago!.

So I’m not willing to point fingers at Amazon at this stage.

I don’t think code repos were used in the attack; at least this time there wasn’t a supply chain issue! But the attacker did put her results into github, and that may be where some of the confusion came from. There’s at least one suit being brought against github for allowing this data to be stored there

Senator Rob Wyden’s letter looks a lot like a fishing expedition along the lines of “your customers keep f***ing up; what are you going to do about it?“, with a side helping of “you created something easily broken; you hold some responsibility”.

The thing is that SSRF attacks have been known for a long time. Indeed the OWASP page for them explicitly mentions the metadata service as an example… and that’s from 2017.

Cloud server meta-data - Cloud services such as AWS provide a REST interface on http://169.254.169.254/ where important configuration and sometimes even authentication keys can be extracted

So that adds a new question

Why didn’t Capital One pentesters discover this?

Now as I was writing this up I came across Amazon’s reply to Senator Wyden.

It doesn’t really clear up the WAF question, although it feels to me as if they’re talking about third-party WAFs. They’re definitely putting the blame at Capital One’s feet:

As Capital One outlined in their public announcement, the attack occurred due to a misconfiguration error at the application layer of a firewall installed by Capital One, exacerbated by permissions set by Capital One there were likely broader than intended.

One interesting part of the letter is that they’re going to start scanning their public IP ranges for similar misconfigs, and alert customers. This is similar to other sorts of scanning that Amazon pro-actively do (e.g. searching github for API credentials).

Is this tacit acknowledgment that they are partially to blame? Maybe :-)

My conclusions are:

AWS are not to blame for the breach
- AWS may have made it too easy for misconfigs to happen
Github was not used to create the attack; it just stored the results
This is a pure Capital One f**k up

Summary:

I really am not a fan of the AWS security model; there’s far far too many knobs and controls, and it’s not clear how the interact with each other. It can be hard to even know something simple (“Is this server port 22 open to the internet?“) because of how configurations interact (security groups, routing tables, network ACLs, etc etc).

It is critical that any company using the public cloud understand the technology they are using. Just because Amazon provide 100+ services doesn’t mean you properly understand how to secure them. How many data breaches have occurred because of backups in insecure S3 buckets, for example?

In the Capital One case, SSRFs and the metadata URL are well known. It should have been detected during application security testing. There’s more than one problem, here!

SecDevOps? DevSecOps? SecDevSecOpsSec!

Sun, 07 Jul 2019 14:34:24 -0400

I was at a conference the other week and the discussion turned to DevSecOps, and the comment of “it should have remained SecDevOps” was made. Now I’m a security guy so I joked that it really should be “SecDevSecOpsSec”, which got a laugh.

But I was actually serious because I feel the focus on DevSecOps is causing a lot of other work to be missed.

DevSecOps is good…

A lot of the focus on DevSecOps is around improving code quality and security without harming the productivity enhancements that DevOps has brought to the table.

So now security gets embedded into the coding life cycle. This may be during code check-ins, during your CI/CD build pipeline, even embedded the your IDE itself! Different quality of results can be performed a different times; an IDE checker may only perform basic stuff, but a code merge may kick off a full SAST scan, and the CI/CD process can do full integration and DAST scans.

At the end of the day you have a greater level of confidence that your developer hasn’t created a new buffer overflow or SQL Injection attack vector, that passwords aren’t hard coded into the app… the list goes on!

It’s all good stuff.

… But it’s not enough

The problem, though, is the focus. By concentrating on this code delivery pipeline we start to forget that there’s more to an application than code delivery.

How was this application architected? If you’ve stuck a MongoDB (with default credentials) into your DMZ then you may have the best code delivery in the world, but you’ve delivered an awful result. How is authentication in your application handled? Does the development team even know what all the non-functional requirements are to deploy an app of this type? If you’re writing a PCI app, or a HIPAA app, or dealing with PII data… Are you deploying to Singapore and so meeting MAS requirements? Argentina and their Central Bank? Are you logging properly? Managing encryption keys? What impact does this project have on other projects?

So right at the start of the project, Security Architecture needs to be at the table to ensure these requirements are documented and in place.

Then we need some oversight; if you’re storing data in an S3 bucket then the Security requirements may define how the bucket needs to be restricted. There may be regular validation that the policy hasn’t changed, or that improperly scoped data hasn’t been put in the wrong place (no credit card numbers in a public bucket, please!).

Products live beyond the life of the primary development team; they become “stable” with only minor break-fix changes. Security monitoring lives outside of the DevSecOps loop and acts as a check that configuration and controls haven’t drifted.

And then there’s product sunset (your original project plan did have a sunset phase, yes?). Ensuring that EOL applications are properly handled has more security concerns.

Conclusion.

DevSecOps is a pretty good solution to a subset of the problem space. It can be used to create better code quality, reduce errors (especially security ones), speed up delivery by reducing the iteration cycle time, and reduce friction. All good things! And the technology is getting better all the time.

But the role of security starts at project initiation, it lives through the DevOps life-cycle, it monitors the result and keeps on monitoring it even when the product is in “maintain” state, and is there when the product is finally decommissioned to ensure secure data destruction and everything else.

There’s Sec at the beginning; Sec in the middle; Sec at the end.

SecDevSecOpsSec; it doesn’t trip off the tongue, but it’s important!

When MFA isn't necessarily strong

Sun, 09 Jun 2019 12:10:40 -0400

Every day we hear of yet another data breach. One common reason is because of password compromise. The problem may be because of successfully phishing; it may be due to password re-use; it may be due to brute force attacks; it may just be weak passwords.

So it is now considered best practice to use some form of Multi Factor Authentication.

To quickly summarise, MFA is 2-or-more of the following factors:

Something you know. Typically this is a password
Something you have. Maybe a physical token of some form
Something you are. This is form of biometric, such as a fingerprint

You don’t repeat the same factor, so two passwords isn’t MFA.

Factor Strength

Ideally these are also unique; it’s not just “something you know”, it’s “something only you know”, “something only you have”, “something only you are”.

This unique part isn’t a strict requirement, but it feeds into the strength of a factor. A PIN that is only 1 digit long (“something you know”) isn’t very strong ‘cos there’s a 1 in 10 chance of guessing it, and the attacker may have multiple attempts. A 4 digit PIN on a debit card may be considered weak, but we can mitigate some of the weaknesses by limiting the number of attempts.

That debit card I just mentioned? It’s a factor; “something you have”. Ideally only you have it… but the card stripe can be copied so the card duplicated. This is what card skimmers do; they read your card data and store it, allowing for a copy to be made.

We need to consider how strong a factor is and what mitigations can be put in place before deciding whether it’s suitable to be used as part of an MFA solution.

Human factors, of course, play a large part in evaluating the strength which is why passwords are frequently panned; they can be stolen in so so many different ways!

Even strong solutions aren’t perfect.

A very common solution to MFA is a hardware token such as the RSA SecurID.

Typically you also need a PIN to go with the token number; the combination of the PIN+token is the authenticator.

These are a relatively strong form of “something you have” because they are close to impossible to copy; the magic numbers can’t be extracted from the device.

I say “close” because there are ways around it. Is the provider securing the magic number properly? In 2011 RSA suffered a breach that exposed the seed tokens meaning that millions of SecurID tokens were compromised.

But user factors also apply. One I’ve seen, before, is to just point a webcam at it and expose that webcam to the internet. Now you can get to your token from anywhere in the world without carrying it. Of course this is bad security, but people do things to make life easier.

We understand this and accept that some people will be silly.

When similar solutions are totally different.

RSA also provide a “soft token” solution.

From their brochure:

First introduced in 2002, RSA SecurID Software tokens are cost-effective, convenient, and leverage the same algorithm as the RSA SecurID key fob style token. Instead of being stored in hardware, the software token symmetric key is secured on the user’s PC, smart phone or USB device.

This sounds great! All the benefits of SecurID, using the existing authentication infrastructure, and without the cost of shipping around physical tokens. Why wouldn’t you want this?

But is the solution of similar strength?

A softtoken requires a number of inputs to work:

The Device ID. This is used as a form of encryption key to protect the seed.
The token seed. This is the primary material used to generate the tokens
A PIN. This is a shared secret known to the user and the authentication server
The current time.
The algorithm. This is pretty much public knowledge

If you know all of these things then it’s possible to copy this token.

Here is where management of the solution is important.

The RSA SecurID software exposes the Device ID. This is needed to request a token in the first place. The user knows the PIN. The algorithm is known, and the time… well, we better know the time!

So the security of the solution depends purely on the handling of the token seed. And, unfortunately, older solutions (still in use) can tell the user about this via email. It typically looks like a URL of the form

http://127.0.0.1/securid/ctf?ctfData=3D200040994....

The number after ctfData is the important part, and this is protected by the deviceID.

With this it becomes perfectly possible to import the same data into the stoken software:

% stoken import --token http://127.0.0.1/securid/ctf?ctfData=3D200040994...
This token is bound to a specific device.
Enter device ID from the RSA 'About' screen: blahblahblah
Enter new password:
Confirm new password:

(Leave the password blank to not encrypt the resulting config file).

Next we can store the PIN

% stoken setpin
Enter new PIN:
Confirm new PIN:

This information is stored in a simple config file

% cat /home/sweh/.stokenrc
version 1
token 200040994......
pin 12345678

The “token” value isn’t the one from the email; it’s been “unprotected” using the entered Device ID. And, no; that’s not my PIN!

And now the moment of truth:

% stoken
31636434

And that matches the number displayed on my phone. Without needing my phone or entering the PIN or anything else.

This is a simple configuration file that can be copied to 100 different places and all of them will work identically.

That’s the easy way to get the data. There’s always the hard way; compromise the device, steal the contents from the phone’s storage.

So, clearly, SecurID soft tokens have different security characteristics to a hard token. The data can easily be copied. We no longer have a strong “only” on the “something you have”.

We have effectively reduced the strength of SecurID to that of a TOTP solution such as “Google Authenticator” or “Authy”.

But it still may be good enough

You can mitigate the problem to some extent; newer SecurID softtoken solutions can perform auto-registration and hide the token seed from the user (if you don’t see it, you can’t copy it!). You could have an out-of-band management process that generates new seeds every week (to limit the exposure of stolen/lost keys). We can require rootkit detection processes.

There are ways of making it stronger. None are perfect, but security isn’t about perfection.

But even with the weakness of the old system, we have raised the bar to an attacker.

Google did a study that showed that even the relatively weak SMS based 2FA protects against all automated bot attacks and 96% of bulk phish attacks.

On-device authenticators prevented 99% of phish and 90% of targeted attacks.

So is SecurID Softtoken strong enough, even with the gaps I’ve identified? Probably!

Conclusion

MFA is around gaining a level of confidence that a person authenticating as a user is the person authorised to authenticate. If they have a password and a softtoken value then you have more confidence than if they have a password, alone.

It becomes a risk management evaluation as to how much money needs to be spent to raise the bar; going from 0% to 90% is pretty easy; going from 90% to 95% costs a lot more.

You need to decide the cost/benefit point for yourself. No solution is perfect.

As for me, authy works sufficiently for my personal stuff; if an attacker really wanted access to my account then this would probably work!

Adding some smarts to a dumb aircon

Sun, 02 Jun 2019 12:26:27 -0400

I have a Friedrich aircon. It’s of the old school. The only intelligent part of it is “eco” mode (turn off the fan when the temperature is cold enough) and a simple timer (“turn on in 9 hours time”).

It’s this timer that annoys me; you have to set it every day. A number of days I would go to work, forgetting to set it, and come home to a house in the mid-90F.

So I wondered if there was a way I could remotely control it. Now I’ve previously played with an Arduino and the IR library and successfully managed to have it send the POWER signal to the aircon. Unfortunately this is a toggle; if the aircon is already on then the signal turns it off. For remote access I needed to know the current state of the aircon.

I considered a smart plug to measure the power draw, but this won’t work due to the “eco” mode.

The front panel looks like this:

I thought that if I could detect if any of those LEDs were on then I could use this as a proxy for if the aircon was “on” or not.

The hardware

Via a reddit chat, I came across a post around reading a smart-meter LED. This lead me down the path of a photo-resister. You put these across an ADC port and a voltage supply and the resistance of the component changes depending on how much light is on them. And they’re small; about the same size as the LED! That would fit nicely in that area.

So I tried out a simple design, using a photo-resister, a pull-down resister and a spare LED blaster I had from previous tests and hooked them to a NodeMCU ESP8266.

And it worked; if I covered the photo-resister then the value read from A0 dropped to zero; if I shone a light on it then the value shot up.

So now I needed to fit it all into the control panel space. I decided to use some doorbell wire left over from the garage project. Only the IR transmitter and the resister needed to be inside the panel:

I cut a hole in the side of the door to route the cable through. I’m not so happy with this; when I open the door it drags on the cable and moves the photo-resister. I may need to make the hole bigger.

Nicely there was a small gap along the top of the control panel which the doorbell wire fit into!

On the ESP side I connected the wire to some cables that let me put it on the legs. I didn’t want to solder the cable to the board because I wanted to be able to disconnect it for reprogramming. I did solder the pull-down resister to the board, for simplicity:

The software

So now I have a network accessible board capable of reading the on/off state (OK, the state of the “Eco” LED!) and send IR signals to turn on/off the aircon. The rest is just a simple matter of programming.

Essentially I make the ESP send the current state of the sensor on an MQTT queue once a second. On another queue it can receive commands (“POWER ON”, “POWER OFF”, “POWER”) and depending on the state of the sensor it can decide whether to send an IR signal or not. I had to use a rolling average of the sensor readings because point readings weren’t too accurate. But it works!

Alexa integration

It’s now pretty easy to add a fake Hue bulb. A loop can read the status channel

#!/bin/ksh -p

# Dining Room Aircon

STATUS=${1:-/tmp/aircon}

temp=0
/home/sweh/bin/mqttcli sub --host hass -t aircon/EA42DF/status | while read line
do
  kill -0 $PPID 2>/dev/null || exit

  print $line > $STATUS
done

So now the main program just has two simple functions:

get_aircon()
{
  SWITCH["DiningRoomAirCon"]=$(cat $AIRCON_STATUS)
}

set_aircon()
{
  /home/sweh/bin/mqttcli pub --host hass.spuddy.org -t aircon/EA42DF/control -m "POWER$1"
}

At this point I can now say “Alexa, turn the dining room aircon on”… and it works!

And the first thing I did at that point was to add a “power off” to my bedtime routine, because I’ve lost count of the number of times I’ve gone to bed forgetting to turn the aircon off and been woken up as the compressor kicks in (I’m a light sleeper).

Home Assistant Automation

HA integration is also simple; just point to the message queues:

switch:
  - platform: mqtt
    name: "Air Con"
    state_topic: "aircon/EA42DF/status"
    command_topic: "aircon/EA42DF/control"
    state_on: "ON"
    state_off: "OFF"
    payload_on: "POWER ON"
    payload_off: "POWER OFF"

Plans and limitations

There’s still a limitation with this; I don’t know what the temperature is set to. But since I rarely change this (68F setting appears to make the living room around 78-80F, which is comfortable) I’m not too worried.

More interesting is the ability to now control this via home assistant. I already have a reading from my smart thermostat, so I know how warm the living room is. It’s possible I could set up an automation for something like “if the living room is over 85F then turn on air-con”. I’d probably time-bound this so it only does it between 2pm and 9pm. Also need to work out how to not send the “ON” signal every 30 seconds :-).

All possible and feasible.

The result may not be a smart aircon, but it’s not as brain-dead as it was before!

Extending automation to the garage

Sun, 19 May 2019 12:03:45 -0400

My garage door is controlled my a Lynx 455 Plus garage opener

This is a pretty traditional opener; a door-bell type button inside the garage to open/close the door and a remote control for wireless access.

I wanted to see if I could make this smart-enabled. Now the control side is simple enough; just put a relay in parallel to the button. If the relay closes then the opener will think the button has been pressed.

It would also be nice to have a sensor to detect if the door is open or not. For this a magnetic reed switch would work. Mount the switch to the frame and stick the magnet to the door. When the door is closed the switch activates (e.g. closes or opens, depending on if the switch is “NC - normally closed” or “NO - normally open”) and when the door opens the magnet moves away from the reed switch and the state changes.

So the concept is easy, but what we can use to put all this together?

Enter the Shelly 1.

This is a small device. In some respects it’s a competitor to Sonoff switches but has some nice features. In particular it has a “switch sense” line. By default this controls the relay, but in the latest firmware that can be disconnected and acts purely as a sensor. Nicely this device can also be powered from low voltage DC and the garage door opener has 24V DC available, so this looked perfect for my needs.

Spoiler: it turned out that the 24V from the opener wasn’t stable enough to drive this, so I had to run a power line anyway. But everything else went to plan!

For Sonoff fans, the Shelly 1 can be flashed with Tasmota firmware, but I didn’t feel the need. The latest firmware did everything I need out of the box.

The Shelly 1 can also be connected to a cloud for remote control, but I didn’t want this so I never enabled it. Indeed I wanted other functionality (MQTT) which prevents cloud action. So all control is local.

Setting up the Shelly 1

On first powerup the Shelly works as a WiFi access point. From here connect a device (I used a Chromebook, but a phone would work), browse to the configuration page, tell it to become a WiFi client of your normal WiFi network. And that’s all it takes to make it reachable. I also added a username/password requirement. It’s only “basic auth” over a http connection, so it’s not very secure, but at least it stops accidents.

For the garage door opener use case I configured the “Settings” page so that on power-up the device defaults to OFF. I set the button type to “Detached switch”.

On the “Internet and Security” tab, under advanced settings I configured MQTT to point to an MQTT server on my HA machine. Note that the server has to be specified by IP address (eg 10.20.30.40:1883) and can not be SSL enabled. Fortunaly the common “mosquito” server works just fine, and is common in HA setups.

In case it helps, I also set up the following:

Min reconnect timeout: 2
Max reconnect timeout: 60
Keep alive: 60
Clean Session: y
Retain: n
Max QoS: 0
Will Topic: shellies/shelly-garage/online
Will Message: true

On the “Timer” tab I set “Auto Off: When ON Turn off after 2 seconds”. What this means is that when the relay is triggered then it will auto turn off after 2 seconds, so it acts like a press and release of the button. At least that’s how it will look to the door opener!

Monitor and control

Now with this running we should be able to see regular output messages on the MQTT server. The exact queue name will depend on the device ID. In my case I can see

% mqttcli sub -d --host hass -t shellies/shelly1-24DCEF/input/0
INFO[0000] Broker URI: tcp://hass:1883                  
INFO[0000] Topic: shellies/shelly1-24DCEF/input/0       
INFO[0000] connecting...                                
INFO[0000] client connected                             
INFO[0003] topic:shellies/shelly1-24DCEF/input/0 / msg:1 
1
INFO[0008] topic:shellies/shelly1-24DCEF/input/0 / msg:1 
1
INFO[0013] topic:shellies/shelly1-24DCEF/input/0 / msg:1 
1

So every 5 seconds there’s a message showing the state of the switch. In this case the switch is closed. If the switch was open then it would show a 0. Interestingly a state change causes a message to be sent immediately.

We can also control the relay by sending a message to the API

curl -X POST -u user:pass http://shelly-garage/relay/0 -d 'turn=on'

(“shelly-garage” is the DNS name I set for the Shelly). This should turn the relay on (you’ll hear it click) and then the timer should auto-turn it off after 2 seconds.

It’s also possible to control the relay by publishing to an MQTT queue, but I didn’t do that.

Alexa integration

Now we’re in a position to start controlling things!

Because I wanted to control this from Alexa, I used my fake Hue software to make it pretend to be a bulb.

Reading the switch status was done with a background process:

#!/bin/ksh -p

# ON means Garage is open

STATUS=${1:-/tmp/garage}

temp=0
mqttcli sub --host hass -t shellies/shelly1-24DCEF/input/0 | while read line
do
  kill -0 $PPID 2>/dev/null || exit

  if [ $line == 1 ]
  then
    onoff=OFF
  else
    onoff=ON
  fi
  print $onoff > $STATUS
done

I did it this way so the main loop just needs to read the /tmp/garage file to get the state, with no delays.

Now there’s no true open/close control for this door; you just press the button. So for simplicity I just make on/off call the same function. The routines used in the media loop look like this:

get_garage()
{
  SWITCH["GarageDoor"]=$(cat $GARAGE_STATUS)
}

set_garage()
{
  curl --max-time 2 -X POST -u user:pass http://shelly-garage/relay/0 -d 'turn=on'
}

With this fake switch discovered by Alexa I could now say “alexa turn garagedoor on” and the door opens. Similarly “alexa turn garagedoor off”.

It’s easy to create a routine now: “Alexa open the garage door” and even “Alexa open the pod bay doors” just calls the smart home bulb on function.

Monitor and alert from Home Assistant

I thought it would be nice to have an alert whenever the door was opened or closed. Enter HA.

First we need to set up a sensor to read the state of the switch:

sensor:

  - platform: mqtt
    name: "Garage Door"
    state_topic: "shellies/shelly1-24DCEF/input/0"
    value_template: >
      {% if value == "1" %}
        closed
      {% else %}
        open
      {% endif %}

This creates a sensor called sensor.garage_door and has a state of “open” or “closed”. In my case my reed switch returns a “1” when the door is closed. If the switch you use returns a “0” then just change this line.

With this we can add a badge to the standard Lovelace UI

    - sensor.garage_door

An automation can now be triggered on the change of state of the door

- alias: Detect Garage
  initial_state: true
  trigger:
    platform: state
    entity_id: sensor.garage_door
  action:
    service: notify.alexa_media
    data_template:
      target: 
        - media_player.living_room
        - media_player.kitchen
        - media_player.stairs
      data:
        type: announce
      message: >
        {% if trigger.to_state.state == "closed" %}
        "Garage Door Closed"
        {% else %}
        "Garage Door Opened"
        {% endif %}

This makes use of the alexa_media component.

Physically setting it up.

All the work, so far, can be done on a bench.

Basically the wiring is as follows:

The 0/1 lines should be connected, in parallel, to the existing button push. It doesn’t matter what way round the wires go.

The SW and L lines should be connected to the reed switch. Again it doesn’t matter what way round the wires go.

Finally the L and N should be connected to the power. Since we’re using low power DC the L should be connected to ground and the N connected to the + voltage. This sounds backwards, but it’s not.

My original intent was to use the 24V power line from the door opener, but when I tried the Shelly just didn’t power up at all. So in the end I used a 12V power supply; cut the plug off it and connected the cables to the terminals.

Future state

Going forwards I would like the HA notifications to be location aware. For example if I’m away from home and the garage door is opened then ping my phone. Or if I’m leaving home and the garage door was left open then alert me. I haven’t yet found a good location tracker for HA that works with my Android phone, so that’s still a work in progress.

Slowly making my home smart

Sun, 19 May 2019 11:18:10 -0400

This post isn’t in my normal theme. I’m gonna describe how I made my home smart. Well, semi-smart.

Over the past couple of years I’ve slowly been making my house lights be smart. In many places I’ve used Hue White Bulbs. They’re frequently on sale and can be got for around $10/bulb, which isn’t bad. With the hub they can be controlled by Alexa, or locally by using the API exposed on the hub.

Where I can’t use Hue, I’ve been using Lutron Caseta switches. These can work without a neutral wire, so are a great retrofit for older houses. It also has a hub that works with Alexa, and some smart people have worked out how to talk to the hub with python libraries.

I was able to extend the use of Alexa to other areas by emulating Hue bulbs. So now when I come home I can say “Alexa I’m home” and it will turn on the living room lights and start iTunes playing on the media center (turning on stuff that’s needed). Other routines (“Alexa I’m going out”, “Alexa bedtime”) do other useful stuff.

I replaced my thermostat with a Radio Thermostat CT50. This has a local API, so with my “fake hue” software I was also able to add basic Alexa controls.

But this is just the beginning.

Enter Home Assistant. This is an open source program written in python. It can be run in docker, on a Pi, as a VM, or inside a python venv. I chose the latter approach; created a CentOS VM (chosen so all my management tools work with it) and deployed HA.

HA can integrate with Hue via the API, with Caseta via the previously mentioned python libraries. Someone even worked out how to talk to the Alexa service (although it’s really not production ready). This lets me create a reporting interface such as

Or a simpler “light” panel.

These panels are reactive; if a light turns on then the corresponding icon lights up. You can also control the lights from here; click on a light and it will turn on/off the real light.

OK, so far so controllable. It’s providing interesting and new ways of looking and managing the environment. But there’s no smarts.

Enter the “automation” part of HA.

Firstly, my basement has two sets of old-school florescent bulbs; one by the stairs and controlled by a switch outside the basement, and one in the back (where I have my computers set up), controlled by another switch. I wanted to treat these lights as one. So if I turn on the basement light then the one by the computer also turn on.

With HA you can write a trigger:

- alias: Basement Automation Turn On
  initial_state: true
  trigger:
    - platform: state
      entity_id: switch.basement_computer_lights, switch.basement_main_lights
      from: 'off'
      to: 'on'
  action:
    service: switch.turn_on
    entity_id:
      - switch.basement_computer_lights
      - switch.basement_main_lights

This basically says that if either of the Caseta light switches are turned on then make sure they’re both turned on. There’s also an equivalent “off” function (not shown); if one is turned off then both are turned off. Now HA doesn’t get instant notification of changes because it has to poll the hub, but it generally fires within 3 to 5 seconds, which is good enough.

We can also use this idea to cross environments. In my laundry room I have two Hue bulbs and a motion sensor. When I enter the room the motion sensor detects it, checks the lights levels and turns on the bulbs if necessary. This is out of box Hue functionality. But we can set up an automation that detects the bulbs turn on and also turn on the florescent bulb in that area. This one is a bit more complicated because I’ve combined on/off functionality and the Caseta switch detection into one routine.

- alias: Laundry Automation
  initial_state: true
  trigger:
    - platform: state
      entity_id: switch.laundry_main_lights, light.laundry
      from: 'off'
      to: 'on'
    - platform: state
      entity_id: switch.laundry_main_lights, light.laundry
      from: 'on'
      to: 'off'
  action:
    - service_template: >
        {% if trigger.to_state.state == "on" %}
        switch.turn_on
        {% else %}
        switch.turn_off
        {% endif %}
      entity_id: switch.laundry_main_lights
    - service_template: >
        {% if trigger.to_state.state == "on" %}
        light.turn_on
        {% else %}
        light.turn_off
        {% endif %}
      entity_id: light.laundry

The result is that I can walk into the laundry room and all 3 lights turn on, and a few minutes after I leave the hue motion sensor turns off the Hue bulbs and all the lights go out. Or I can manually turn on/off the florescent bulb from the Caseta switch and the Hue bulbs will follow.

This is just the beginning of making my home smart. Just with what I’ve got I could set up triggers based on, for example, sunset. In most cases I’m happy to have trigger based routines (‘Alexa foobarbaz’) but it’s nice to have the laundry lights work when my hands are full with a basket of dirty clothes.

Which reminds me… I need to do laundry!

Emulating a Philips Hue light

Mon, 19 Nov 2018 12:42:40 -0500

Over the past couple of years I’ve been building out an Alexa skill for my media center. So now I can say things like “Alexa, tell media to play music”. That will turn my receiver on, switch it to the Mac input, and start iTunes playing. Similarly “Alexa, tell media to switch to TiVo”, “Alexa tell media to pause”.

The backend code running on the Mac asks the receiver what input is selected (TiVo, Mac, BluRay…) and if on the Mac it works out what application has focus (iTunes, DVD Player, Kodi, …) so when a command such as “pause” is received then it knows how to send the appropriate action.

This works pretty well. But there’s a problem. Skill actions can’t (currently, anyway) be used in routines. So I can not say “Alexa, I’m home” and have it turn on the lights and start playing music.

However, home automation devices can be used in routines. Since I have a Philips Hue setup (“Alexa lights on”) I wondered if I could create fake lights and have those lights map to actions.

Looking around I found a number of Hue emulators; some in python, some in perl, Java, Go… However they were all “monoliths”. They came with a core library and then you added your own routines (“create this light”). None of them really worked the way I wanted.

My plan was to have a “bridge” routine. This would handle all the low level API layer stuff (UPNP discovery, Hue API, etc) but wouldn’t do any real work itself. Instead there would be a child process that would do the hard work. This could be written in any language (e.g. bash) and would communicate with the bridge via stdio, using a very simple set of primitives.

So now when the virtual light was turned on the bridge code would send a “Turn on” message to the child, which can take any action it needs to, and send a response back. The bridge caches the response so that status requests can be handled quickly, but the child can send updates whenever it likes.

This now allows the real world to be modelled. If I turn the volume up on the receiver then the child can send an updated “brightness” value to the bridge, and anyone using the Hue API to get the status will see the new value. Similarly lights could be linked; if I switch to BluRay input then that virtual light would be turned on, and the Mac, TiVo lights automatically turned off.

The Alexa app makes it seem like a normal bulb with brightness

The resulting code is HueBridge. I wrote the bridge code in Go. There’s a simple shell script example (test-media) that shows how this might be used to control a media system.

Using this code I’ve also created a virtual “Christmas Music” light. So in a month’s time I can say “Alexa, it’s Christmas!” and it’ll turn on the tree lights (via a TP-Link power switch) and start playing Slade’s “Merry Christmas Everyone”.

Now that’s a good use for Smart Home technology!

Career advice

Mon, 08 Oct 2018 09:43:12 -0400

I get email…

What are your thoughts about making a career out of specialising in Unix? It seems like you’ve done quite well…

Interesting question…

Realise that I started doing this 30 years ago. At that time there was no Windows (Windows 1.0 was around the corner). We had DOS. Networking was mostly serial based; if you were (un)lucky you might have had Banyon Vines or Novell or some other proprietary network stack.

Since my course work was hosted on Sun machines and the Microcomputer Society had access to the Physics Dept Unix machines, I kinda fell into Unix my default. So I taught myself DOS, PC hardware… and Unix.

If the coursework had been on the Vax cluster then I might have been a VMS person instead :-) [ I just built a virtual cluster, using simh so I can see what I missed out on! ]

I got lucky; in particular the TCP/IP networking stack finally got ported to the Microsoft world so I didn’t really miss anything by avoiding all that messy stuff!

But what I also learned was the lower levels; cabling, networking, telecoms, databases, appdev, security…

So I’m a generalist, but with a Unix speciality focus. After all, I’ve been doing Unix for 30 years; I can do a lot of it in my sleep :-)

Now, for you, it depends on what you want to be.

Looking forward, I can see Unix (Linux, really) remaining a core product on the internet. I read a story recently (I can’t find the link) that said that even on Azure, half of the VMs running are Linux based. On AWS, of course, it’s the vast majority.

If you want to be a “full stack” developer, understanding Linux is very beneficial; it’ll let you tune your app, understand why things work the way they do.

If you want to be a server sysadmin then Linux is the way to go; Windows servers are great for supporting windows workstations but I can see that become less important as things go cloudy (why run Active Directory servers, when Microsoft can do it for you? Why run fileservers when… When all your desktop apps are cloudy SaaS offerings in Office365, who needs on-prem servers?). But sysadmin will be more “Infrastructure as Code” related, so learning to script and automate will be essential.

Desktop side is going to be Windows for a long time, even with the cloud; they may be virtual desktops running in Azure, but they’ll still be there, so will have all the requirements around management (antivirus, privilege control, etc etc). Personally I find that tedious, but there’s a lot of engineering work going into taking the Microsoft product and making it suitable for Enterprise deployments.

If you want to be an architect then spreading your awareness out of one area and looking at others will help. Or you could look at stuff higher up the stack (eg Identity and Access Management; firewalls; intrusion detection/prevention; monitoring; vulnerability scanning) which may cross technology towers.

Ugh, this got a bit long…

Summary: I don’t think you can go wrong, learning Unix. This can be the foundation and core skill set. But it shouldn’t be the only skill. The more you know around how things fit together the better a technologist you’ll be, and the more valuable your core skills become. “I can do this with Unix” is great; “Here’s a problem, I know how to put A, B, C together and solve it” is even better!

Hope this helps :-)

When Development is Production

Sun, 23 Sep 2018 18:56:06 -0400

It’s an article of faith that the development process starts in the part of the network set aside for development work. Then the code may go to the QA area for QA testing, UAT area for UAT, production area for production.

That statement almost looks like a truism; development work is done in DEV.

So a corporate network may be divided along dev/uat/prod lines, with firewalls between them so that development code can’t impact production services. After all, we don’t want some development code that’s doing application schema updates to accidentally drop the production database!

But this is not always true.

Development is production

For the development team, themselves, they have expectations on the platform infrastructure to be available so they can do their work. If the VMware team decides to test a new patch set in the DEV environment and brings down the development cluster then they’re gonna have a lot of annoyed developers banging on the door. Depending on the severity of the outage this could cause application delivery timelines to slip, for contracted delivery dates to be broken and cause actual financial impact to the company.

Further, some of the core infrastructure may be shared between environments. The SAN arrays may have LUNs assigned to DEV machines and to PROD machines; networking switches (due to VLANs and trunks, inter-datacenter links) may share paths.

So if you’re going to do something that may impact core infrastructure or large numbers of development machines, you don’t start in DEV… you start even earlier.

The lab is for more than use Proof of Concept tests

A lot of people think of “lab environments” as places to do tests. “Hey, we want to test out some piece of software, see if it does what we want. Let’s spin up a proof of concept in the lab.”

And, yes, that’s the right place to do it. Labs are typically isolated from the main networks and needing jumphosts or bastion hosts to be accessed in order to get to the machines. They will have separate network infrastructure, separate Active Directory domains. They’re probably not as well managed and domain admin privileges may be easy to get. This leads to some levels of instability.

But despite this, the lab is also where sustained engineering processes start.

Sustained engineering

Products have a life cycle. They get introduced to the company, tested, deployed, updated, patched, and then (if you’re lucky) sunsetted and shut down.

The introduction is in the lab, as part of the aforementioned Proof Of Concept. Then rolled out to DEV…

But the lab testing doesn’t stop there. Updates should also start in the lab. Let’s say you want to test a new version of the anti-virus package that you’ve been using for 4 years; this should start in the lab. You want to test a new data collection command to be deployed to the existing tooling; this new command should start life in the lab. You want to a new new ansible/chef/puppet/cfengine rule? Start in the lab.

What this means is the lab is a persistent environment. You may destroy and rebuild components of it so the thing doesn’t diverge too far from what is deployed to the main networks, but you’ll always have your AV suite, your data collection tools, your automation tools in the lab.

You then need to deploy to the rest of the network, which may be done in a risk-managed phased approach.

For example, you may not have all the tools available in the lab to talk to (HR systems, for example, may not be there). So you need to perform integration tests outside of the lab. This should be in an “engineering QA” environment; effectively your area of the DEV environment, made to look as close to production as possible. Only when these tests have passed can you start to deploy across the environment…with caution.

You need to look at what impact an outage may have; clearly production impact is worst, but is the DEV environment more/less impactful than QA/UAT? One may be customer facing… but may not be used so often. You may also want to look at regional sizes; perhaps deploying to APAC/DEV first will let you see how things work in the “real world” but having least impact. Then EMEA/DEV, then AMER/DEV…

Once that’s settled in (maybe a week) you could do {APAC,EMEA,AMER}/UAT… and then a week later, {APAC,EMEA,AMER}/PROD. Any deployment issues should be sorted by the time we get to PROD!

Summary

I look at these DEV/UAT/PROD segments this way:

DEV is a production environment supporting application development
UAT is a production environment supporting application testing
PROD is a production environment supporting application production

The key is that all of these are production. If you are providing services to these environments then they are not your DEV/UAT area; these are your production areas, and you need to treat them as such. If you’re bringing in new technology that can impact large numbers of machines (e.g. a new data collection tool, a new automation tool) then you don’t do your testing in DEV; that’s your production environment.

Don’t break production… whether it’s supporting DEV or not!

Privilege Escalation in Unix

Sun, 26 Aug 2018 15:31:16 -0400

In a well controlled environment you typically do not want people logging into servers with privileged access (absent of additional external processes, such as a keystroke logged session manager). If you have 5 SAs all logging in as root then how can you audit activity and determine if it was Tom, Dick or Harry that rebooted the server? Similarly you don’t want DBAs directly logging in as oracle; how do you know who dropped the production table?

Typically this is solved by having the SA (or DBA, or AppOp, or…) login as themselves and then use a privilege escalation method. This method can use alternative authentication processes (e.g. 2FA), audit activity and, if properly configured, limit activity.

On Unix the standard tool is sudo. Some vendors may provide alternatives (e.g. Centrify has dzdo, Fox Technology’s Server Control has suexec) but the principles apply.

Business As Usual vs Elevated privileges

Any discussion of privilege escalation needs to consider the user’s work role. What an organisation considers privileged may not be the same as the operating system.

One simple example of this is the df command. If a normal user runs this then they may not see every file system mount.

$ df | grep dir1

$ sudo df | grep dir1
/dev/shm          249720       0    249720   0% /tmp/dir1/mountpoint

This situation may occur because the mountpoint may not be reachable from the normal user

drwx------ 3 root root 4096 Aug 26 15:42 /tmp/dir1

So from a Unix perspective, we need elevated permissions to run the df command and get a full listing. But from a business perspective it would seem reasonable to allow all the SAs to run sudo df.

The distinction between what the platform considers privileged and what the business considers privileged really needs to be kept in mind, especially when formulating policies and procedures around how to manage this stuff.

Read-only vs Read-write

A number of commands may be obviously read-only; cat is a great example of this; it can read files, but not write to them.

Side Bar: I’ve seen people ask for sudo echo because they think they can do things like sudo echo hello > /etc/my.file. That’s a misunderstanding of how the shell evaluates things; the > redirection happens at the current shell level, with the normal user privileges. Similarly sudo cat myfile > /etc/my.file doesn’t work; the cat may have elevated permissions, but the > redirect doesn’t.

Some commands can operate in both modes. For example miitool and ethtool can be used to see the state of the network adapter, including the negotiated speed and duplex. However it can also change the adapter settings, force renegotiation or even break network connectivity.

Similarly, on some OS’s, ifconfig will only show the MAC addresses if run as root. But now the SA has the ability to change IP addresses and other settings on the fly.

Typically commands that can make changes should not be part of a BAU profile because they may require a change control process before they can be used.

However, commands that can operate in read and write mode may be able to be limited; ifconfig, for example, could be limited to sudo ifconfig -a which would allow the SA to see the MAC addresses but not change things; sudo fdisk -l may be permitted because it allows listing of the partition tables, without making changes.

A number of commands can be limited to read-only access, and so may be considered part of the BAU scope.

Read-only may also be bad!

At a first glance, it might make sense to be able to grant lots of read-only access commands to people. After all, an SA may need to read a root-only configuration file. For example…

$ cat /etc/securetty
cat: /etc/securetty: Permission denied

What would be the harm in allowing the SA to cat every file?

Unfortunately there may be files on the system that should be protected from the SA. Should the SA be allowed to read /opt/my_app/etc/super_secret.txt ? Unlimited cat would allow this.

Unlimited read access may also allow a pivot to another server; for example if there is an ssh key used by a process to send data to another server then this key is probably passphrase-less (it’s a common scenario). Allowing unlimited cat would allow the SA to take a copy of the key and then try and access the remote server.

Obviously we may have audit logs of the abuse of this privilege, but prevention is always better than detection!

Wildcards

We need to be aware of how sudo parses commands. The cat example might be mitigated by only allowing read access to files in /etc;

%sa_group ALL=(root) /bin/cat /etc/*

Unfortunately…

$ sudo -l
User sweh may run the following commands on test1:
    (root) /bin/cat /etc/*

$ cat /tmp/dir1/secret.txt
cat: /tmp/dir1/secret.txt: Permission denied

$ sudo cat /tmp/dir1/secret.txt
Sorry, user sweh is not allowed to execute '/bin/cat /tmp/dir1/secret.txt' as root on test1.spuddy.org.

$ sudo cat /etc/../tmp/dir1/secret.txt
hidden

$ sudo cat /etc/subgid /tmp/dir1/secret.txt
hidden

In this case I was able to make use of how filenames work to display the protected file, and just to use two filenames (one permitted) to match the pattern. Two problems with one configuration entry!

If you plan on using wildcards then determine if that will allow for excessive privileges this may expose. In some cases this might be mitigated with the ! configuration, which will deny commands.

Shell escape

Some commands allow for the user to drop into a subshell to perform activity and then exit back to the program. A common example of this is the less command.

$ sudo less test
myfile
!sh
sh-4.2# id
uid=0(root) gid=0(root) groups=0(root)

Now while the less command itself was logged, all activity performed inside the subshell isn’t properly tracked. The SA has full root level permissions on the machine and we can’t audit their activity.

This may, sometimes, be mitigated with the NOEXEC option in the configuration:

$ sudo -l
User sweh may run the following commands on test1:
    (root) NOEXEC: /bin/less

$ sudo less test
myfile
!sh
!done  (press RETURN)

This isn’t infallible, however, and may block functionality the application needs. It may also not work on every operating system.

Sometimes the most unexpected of commands allows for a shell escape; the one that surprised me was the Solaris format command.

Open additional files.

This is related to shell escapes, but allows for files to be read or modified unexpectedly.

For example, we might want to allow for vi to be run on a single file, but the user can open additional files.

$ sudo -l
User sweh may run the following commands on test1:
    (root) /bin/cat
    (root) NOEXEC: /bin/vi /home/sweh/test

We’ve NOEXEC’d this, but the user can still do :r inside the session to read protected content. (This is hard to screen shot…)

$ sudo vi /home/sweh/test
myfile
:r /tmp/dir1/secret.txt
myfile
hidden
:w! /tmp/dir1/secret.txt
"/tmp/dir1/secret.txt" 2L, 14C written
:q!

$ sudo cat /tmp/dir1/secret.txt
myfile
hidden

sudoedit

Editing of files is a common requirement, so sudo adds another option, called sudoedit. This can be invoked as sudoedit or sudo -e. The way it works is clever; it makes a copy of the file you want to edit in a temporary directory, then runs your editor as your real user, and if changes have been detected copies the file back (with the correct permissions, ownership, etc). In this way any shell escape or file reads are done as the real user, and not privileged.

There are other limitations on sudoedit (eg not in a writeable directory) but it can handle a number of cases where BAU write-file access is permitted.

Functional access (passwordless)

I’ve mostly been focusing on human level access (I’ve described SAs, but this would also work for DBAs, AppOp, and similar, and the target user need not be root).

Sometimes there may be a need for a functional (or service) account to need to perform privileged activity. Examples abound, but a simple one might be a polling look that wants to see what servers have listening ports, and what processes these are. Consider this job being called from cron every hour:

for host in test1 test2 test3
do
  ssh svc_acct@$host sudo netstat -anp | grep -w LISTEN
done

We need root here because of the -p option to netstat.

Obviously we can’t enter the password each time (indeed there may not be a valid password, because we use key based authentication). Fortunately there’s a NOPASSWD: option

$ sudo -l
    (root) NOPASSWD: /bin/netstat -anp

$ ssh test1 sudo netstat -anp | grep -w LISTEN
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      859/master          
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      759/sshd            
tcp6       0      0 ::1:25                  :::*                    LISTEN      859/master          
tcp6       0      0 :::22                   :::*                    LISTEN      759/sshd

These NOPASSWD: entries can also be applied to human user accounts if you consider the activity to be sufficiently low risk that the human doesn’t need to re-authenticate (the df example from earlier may apply).

Block Lists don’t work – Example of a bad setup

I was working with a team some time back who wanted to perform centralised data collection and some of the data required root level permissions. Now, naturally, I refused them ALL access; having a tool with access to every server and having unlimited root on all those servers was an unacceptable concentration of risk.

They went away and came back with the following request:

#List of commands svc_acct is not allowed to execute via sudo
Cmnd_Alias SVCNOEXEC=
adduser*,
chmod*,
chown*,
cp*,
...and so on

svc_acct ALL=NOPASSWD: ALL, !SVCNOEXEC

(In all there were 30 commands listed in the SVCNOEXEC alias).

Apart from the fact this wasn’t a valid sudoers file, these block list setups provide ZERO security in the environment. I was able to think of three approaches within a few minutes:

First obvious way to bypass those restrictions:

Run the command under /bin/sh, which wasn’t restricted:

sudo /bin/sh -c '<any_command_in_that_list>'

sudo /bin/sh -c 'halt'

Second less obvious way to bypass these restrictions:

copy the file to a tmp directory and run that:

cp /sbin/halt /tmp/myprogram
sudo /tmp/myprogram

A third way

Create a wrapper and call that

echo halt > /tmp/myprogram
chmod 755 /tmp/myprogram
sudo /tmp/myprogram

I could probably come up with more esoteric variations as well (symlinks, ld.so, perl, python… even use sed to update the sudoers file, itself!)

Basically, block lists like this can be bypassed with zero effort and should not be used.

Conclusion

The secure way of setting up a sudo profile is to default-deny and then only permit the minimum necessary commands.

These commands should be restricted with options to ensure they are read-only and do not allow further privileges. So, for example, on Solaris sudo format may not be permitted because you can shell out of it. Similarly hbacmd may not be permitted without restriction because it can be used to reset HBAs, change binding rules, etc. but hbacmd ListHBAs is OK.

Of course, understanding business requirements and not blocking efficiency also needs to feed into the decision of what commands you allow an SA to run BAU.

You may decide that sudo cat is OK, trusting on the audit log to be a sufficient activity trail (perhaps with reporting triggered from it if a restricted file is accessed). That’s a definite productivity gain!

Also you may be able to use additional tools; e.g. SELinux maybe able to lock down access to secret.txt, even to users running sudo cat.

If you have any additional guidelines for sudo entry creation, please leave a message in the contents! I know I haven’t covered every scenario (e.g. wrapper scripts) because some of those could be a post in themselves!

Notes from the service for my Dad

Tue, 07 Aug 2018 23:19:02 -0500

I’ve written about Dad’s service.

In preparation for this, Mum and I had written down a bunch of notes for David (the priest). Well, Mum did most of it; I added a few things and a few jokes. David asked a few more questions, to fill in some missing background. This formed the main content of the ceremony.

What I’ve done, here, is taken both sets of notes and put them together, with some editing and tweaking.

This is the story of my Dad.

Douglas Charles Harris, born 19th March 1938 in Dagenham; 2nd of 2 children. His older sister, Rose, passed a number of years earlier.
During the war he was evacuated to Ilkley, in Yorkshire, but after 6 months the family came back to London.
He went to school in Albion Road, Dagenham.
He met Stella in October 1958; she’d just left school and he’d been demobbed from National Service. The relationship got off on a misunderstanding as to where Stella lived; Western Avenue or Western Green. Sadly for Doug, Stella lived further away than he thought so he had a long journey back home!
They got engaged when Stella was 17. Doug changed jobs from being a butcher at Sainsbury’s to work at Fords, so he could save money for a house.
They married on 22nd July 1961 at St Peter and St Paul, Dagenham. People said it wouldn’t last… but here we are, 56 ¹⁄₂ years later!
They bought their first house in Dagenham. Stella surprised Doug 2 years later by putting it up for sale while he was at work. The house sold the same day.
They moved to Limburg Road, Canvey Island. Doug worked in Basildon, Stella traveled to London.
1968 Stephen was born. Never was a man more proud than Doug. Then to their delight Jason was born in 1970.
The family spent many happy years camping on the Isle Of Wight, and in France.
Both boys did well at school. Jason was always popular with his peers and enjoyed sport. Doug and Stella even qualified as referees to support Jason’s football team. Jason gave up football and took up squash, but Doug continued to referee for many more years. He was so devoted to football that he would fall asleep with it on TV at every opportunity.
Stephen was the studious one. He was accepted to attend St Peters College, Oxford. Doug was proud; as he said, he didn’t even make grammar school.
Jason began working in London and soon moved into a flat with a friend. Stephen came back home when he graduated. At last Doug had someone to go down the pub with! Stephen then moved to Hornchurch and Doug had to wait until Christmas for the boys to take him down the pub!
Doug decided to take early retirement at 52 and tried hard to keep busy; golf three times a week, badminton and keep fit twice a week, housework and gardening to fill in the time until Stella came home from work as a teacher, something Doug encouraged and supported.
One time, Doug decided to do some decorating and encourage Stella to go to Spain with a friend. She returned with a lovely present for him… another house, this time in Spain!
Jason married Lorraine, and in 2001 Doug became a totally besotted Grandad to Stuart. This never changed.
Many happy times were spent in Spain, in every school holiday. After Stella retired more exotic places were added to the holiday itinerary, including cruises to far flung places. Always where it was sunny and hot. Doug hated the cold.
This year Doug showed how much he loved Stella; he took her to chilly Iceland for whale watching, then back to Spain to thaw out! And then the trip to Tenerife, to capture the last of the sun.
Stella and Doug had so many lovely memories of the trips they had taken, and had more planned for next year.
Doug was a fiercely loyal man, devoted to family and friends. He enjoyed good food, good company and, after he gave up golf and badminton, he took great pride in his garden. His cry of “It’s Christmas!” will not soon be forgotten!
Tori, Stephen’s partner, remembers Doug as the most welcoming, kindest and funny of men.
He did most things his way, which usually turned out to be the right way. Well… except when he electrocuted himself changing a light bulb. Or when he fell off a ladder, cutting a tree branch. Or breaking the computer. Or…
He will be greatly missed… but the cry of “Stella, I’ve broken the computer!” won’t be.
Doug’s death was sudden, and a shock to everyone. He died in the middle of living; he lived until he died!

Nothing is forever
And only fools would wish is so
Memory is the yesterday that gives us courage for tomorrow

And that, my friends, was my Dad. A sun worshipper. A good man.

In 1992 I had taken a vacation in Florida and bought a cheap disposable camera (which is why these pictures aren’t so good). I had 4 shots left on it, so took these from my bedroom window in 1993. They show my Dad doing what he loved best; tending the garden and sleeping in the sun.

What I did on my weekend

Sun, 24 Jun 2018 16:43:53 -0400

Every so often I get asked to do something that’s not related to my employer, or is stuff that results from my activities for my employer. Frequently this is some form of informal consulting/discussion. There was the cloud expo presentation. I’ve been on a couple of “Customer Advisory Boards” because of my container opinions (I have opinions; sometimes people want to listen to them).

This time I was asked to look at a mobile email configuration. This small company was wanting to move away from their expensive solution to a cheaper, more native, setup. I’d been chatting with one of their security guys down the pub and expressed some reservations, so he asked me to take a look. He created me an account on his PoC project and showed me how to onboard my phone.

How I went about this

Now I’m not a blackhat. I’m not a pen-tester. I don’t have toolkits to try and break stuff.

What I do have, though, is decades of experience in building things. And in making mistakes.

So I look at this as “If I was building this, how would I do it? What mistakes might I make?“. This works out pretty well when it comes to supporting vendor products (if a bug appears I can pretty much understand how it came about and so have meaningful conversations wit the vendor; one of them told me I knew their product as well as their own devs).

It can also help find some of the errors in integration between products; when product A and product B need to be combined to provide a service, there’s a chance for errors.

Now in this case the organisation wanted to use an MDM (Mobile Device Management) to provision to mobile phones and use the native email client to talk to an on-premise Exchange environment. They had worked hard on protecting the gateway (need a certificate to authenticate to the perimeter firewall; username/password to authenticate to the Exchange service). However I still felt this risky; it allowed for more ways of leaking data from the device, itself.

SafetyNet

The first issue was getting my phone registered. I had a Moto X Pure that I’d wiped back to factory defaults… except it had an unlocked bootloader. Apparently this was enough to prevent the app registering, saying it tripped SafetyNet.

SafetyNet is a routine that Google provides that will evaluate your device and report on its trustworthiness. It seems they recently updated the tests to report on the bootloader as well.

Fortunately, MagiskRoot made rooting the phone easy. Since I had an unlocked bootloader I was able to boot into TWRP (not install, just boot), flashed Magisk, rebooted… and now adb shell and su works. That maybe the easiest device root I’ve ever done.

Hiding root

This lead to the next problem… now the MDM app refused to register because it detected root. Magisk has a “root cloak” option. Let’s turn that on for the MDM app. Try again. Failure! Hmm.

Oh hey… the MDM app has a debug mode and an option to send logs to any email address. So turn on debug mode, fail to onboard, send the logs.

Ah! It’s detecting /sbin/su exists and so aborting.

Intereresting, /sbin/su is just a symlink to /root/. It looks like this gets regenerated each boot. The last android device I rooted was 2.4.3; things have changed a lot in the Android security model, and this may be part of the “systemless root” design that’s needed these days. OK… let’s just ‘rm’ it. I can recreate it if necessary since I still have a root window open.

And hmm… The MDM gets to work. Away we go!

I now had a rooted phone and with the Android “work” profile installed and configured.

What can we find

Because this using the native mail client (“gmail”) inside the profile, it’s possible to read the contents of /data/user/10/com.google.android.gm/databases/EmailProvider.db. And, yes, this showed the test emails that had been placed there. Unencrypted, easy to extract. Complete with attachments.

So I’d already proven my point; it’s easy to take an untrusted device, onboard it and extract the data.

Further I was able to see certificates in /data/misc/keystore/user_10. I didn’t bother to try and decrypt them (they’re stored encrypted, but the primary key is decryptable using the device password). In theory this would allow me to connect my Linux desktop to the Exchange environment.

Definitely a way for data to leak the company

But it turned out to be easier

It turns out I wasted a lot of time and effort on this, though. It was a lot easier to export data.

Inside the “work” profile was a copy of the ‘Google Play’ application. This allowed me to install other apps, such as “Word” and “Excel”. It worked well; I could open attachments from email directly in Word and view them. But it also opened a gap; I could connect Word to my personal Dropbox account and then save these attachments to it. Whoops!

Take Aways

The company realised there would be a massive gap in their security posture. I was able to demonstrate two actual paths for data exfiltration (one didn’t even require any specialised knowledge), and one potential. Since “insider attacks” are a big issue, this is something to be concerned about…

They are now looking at enhancing MDM with MAM (Mobile Application Management). The MDM will restrict what applications can be installed to Microsoft ones (Outlook, Word, Excel) and the MAM will restrict what those applications can do (e.g. force encryption at the app layer).

This introduces more complications, but improves their security.

Summary

I’m not a fan of “BYOD” type stuff for email. It really makes keeping your data secure a lot harder. Security of the data is inherently dependent on security of the device and protection of the data stored on that device. This means there a natural tension between the owner the data (the company) and the owner of the device (the employee); the company may want to enforce strong passwords, long PINs, force rotation, disable cut’n’paste… all stuff that may make it harder for the user to use the phone for their own purposes.

This is why “sandbox” applications were popular in the early days; they provided a (semi-)secure environment to hold the company data, while only minimally impacting the device owner.

Mobile operating systems are growing in functionality; Android user profiles are a start in this direction. But they’re still not a sandbox. So using MAM on top of MDM is a necessary component.

I hope I’ll get a chance to kick the tyres on their new setup!

DevOps and Separation of Duties

Sun, 03 Jun 2018 14:01:22 -0400

Reducing the number of personnel with access to the production environment and cardholder data minimizes risk and helps ensure that access is limited to those individuals with a business need to know.
The intent of this requirement is to separate development and test functions from production functions. For example, a developer may use an administrator-level account with elevated privileges in the development environment, and have a separate account with user-level access to the production environment.

PCI-DSS section 6.4.2 “Separation of duties between development/test and production environments”

So here’s a problem. Regulatory environments, such as PCI, require a strict separation of duties between the development teams and the operate teams. This can lead to “Application Dev” (AppDev) and “Application Operate” (AppOp) teams. In some places AppOps may also be combined with Infrastructure Operate (InfraOps), but in others those two operate teams may be separate. The key, though, is that the Dev and Ops teams are separate.

If something goes wrong at 3am then it’s the operate team that’s paged out, who doing primary analysis, call out to the dev team, who may make a fix and have the ops team deploy. In some cases the dev team may not have any access to production at all and are doing their work hands off, which can lead to extended times to repair.

This separation also takes us down the path of requiring heavy hand-over documentation so that AppOp are happy to take on the support. This can slow down release cycles and tends to lead to a waterfall workflow.

DevOps, as a methodology, is designed to break down these barriers; allow faster problem remediation, speed up the release cycle, support agile methodologies.

How can we do DevOps when the Dev team appears to need to be segregated from the Operate team?

Interpretation of the problem

Let’s define what admin access means. In my mind it’s the ability to directly make changes to the environment. An important thing is that the Dev team are permitted to see what is going on in production. They just can’t change it.

How the dev team sees the data is also pretty flexible. It could mean that they can log into the production server and view the logs; they may be able to trace application execution or transactions as they pass from subsystem to subsystem.

Other concerns may impact this, though. Allowing a developer to run a command such as sudo cat /app/log/file may fall afoul of a corporate definition of ‘privileged access’ and so automatically trip a separation of duties flag. It can be argued that any server access is privileged, because even unprivileged users can impact server operations.

One step removed

So rather than give direct access we can focus on a hands-off approach. Tools such as AppDynamics or DynaTrace can create application monitoring dashboards; log collectors (Splunk, ELK stacks) can provide real-time activity logs; apps can be written to allow for introspection. Now it is clear that our developers can get access to the data they need to analyse issues, without going anywhere near the separation of duties issue.

A nice bonus to this is that this it the methodology you want to follow for modern elastic compute environments. It can also help with historical analysis (“what happened 3 months ago”) because the logs are available, which is useful to both the operate and security teams. A central cross-enterprise collector can make life easier for the app teams (no need to stand up unique infrastructure they need to manage) and may provide operational and threat insight, and allow for common development patterns and libraries to be used.

Just make sure the data that you are collecting is properly redacted of any sensitive content (do not push credit card numbers into your log stream!).

But what about changes?

We’ve solved the easy part of merging Dev and Ops; monitoring. But what about changing the environment? Pushing out fixes or new code versions?

This is where automation and CI/CD tools really help.

In theory we could create a pipeline that allows the developer to merge a branch into the “prod candidate” branch; this can kick off a build cycle, automated unit tests, integration tests; it could spin up a QA instance to verify all expected functionality works, and if that succeeds to push to production.

But do we really want that? If we do this then the developer effectively has the ability to change production and so we introduce a separation of duties conflict via a side door.

But the concept is good; we just need some controls on it. Call-outs to an approval process.

An example might be an emergency fix; the DevOps team member may be able to say “release code with tag breakfix-201806031514 under problem ticket 623493”. The CI/CD tool can call out to the problem ticket system, verify it is valid and covers the servers/application in the code scope and perform the deployment. Management oversight performs post-fix reviews.

Similarly a planned release could be done with a similar call, but to the Change Request system, verifying the scope and time window and CR approval state.

A regular cadence may be developed with a “pre-approved regular change”, by pushing the controls further into the dev stack (“merges to the production candidate must have 2 additional developers signing off”).

You are (probably) not Google.

(If you are Google, then you have pretty mature solutions to all this, anyway!)

People look at some of the modern internet giants with their claims of tens of thousands of changes per day and then look at the change model I wrote above and cry “That’s not how it’s done!”. But remember, one size does not fit all. You are not Google; your organisation isn’t structured to be Google. Google is based around rapid iteration; if a change fails it can be fixed with little harm (“Something went wrong; that’s all we know”). Twitter had the “Fail Whale” when stuff broke.

But if you’re handling thousands of credit card transactions per second, what is the consequence of an outage? What will your clients (all those stores who are paying you to process those card swipes) feel? How many of those clients will you lose, and how much future custom will be lost?

Different organisations have different risk tolerances. You are not Google; can you afford this risk?

Exception processes

The problem with automation is that they only handle cases you’ve coded for. You just know that something will go wrong and your AppOp person will need privileged access to the server. This may be a “break glass” type process (“I have an emergency; I need the root password; break glass to get it”) with strong oversight. Initially you may find this process executed a lot, but as your tools, your developers, your procedures mature then this should truly become an exception.

If you still feel the need for an InfraOps team (personally, I think this is a good idea; trained DBAs are better than ad-hoc dev oriented DBAs, similarly for trained SAs) then they may take over the vestigial AppOp requirements that can’t be met by the new DevOps team. Again, engaging these teams may be on an exception basis so doesn’t slow down the DevOps velocity.

Conclusion

What I’m describing here isn’t easy. We’re talking about a requirement that is firmly rooted in the traditional compute method. Organisations have a tonne of embedded processes in place to meet regulatory requirements, and one process may meet multiple requirements so you can’t just skip it. But if you’re looking to pivot towards a DevOps model then investing in the tooling (logging, automation, approvals) can get you a long way towards meeting your regulatory requirements. And nicely set you up for newer technologies (e.g. deployment to a PaaS; management K8S deployments).

SRE is not new

Sun, 13 May 2018 15:38:10 -0400

One of the “hot” things around today is the concepts of Site Reliability Engineering (SRE). I’m gonna be slightly provocative and state that this is not a new thing; we were doing this 30 years ago. Indeed, these concepts go back to where we were when I started out in this industry.

Although, to be fair, there is one new factor.

History

Now I’ll be the first to say that my take on history is very much biased by my personal experiences, and how I worked.

I started in a small company in 1990. I had to deal with the servers, the terminals, the printers, the fax machine, the phone system…. even the door entry system. Pretty much if it was vaguely technology related it became my problem. Plus, of course, “systems development” and creation of tools for the users. Most of the communication was via serial terminals (VT220/320, Wyse85/99) and modems running UUCP. Most of the machines didn’t even have an ethernet port when I started. Naturally this lead to each machine being custom and unique. I learned a lot. Since I was the only person supporting the London office every failure was mine to fix. Nothing like being called during Sunday Dinner to fix a problem because a ship-broker was unable to dialin to complete a deal.

My next job was at an internet publishing division of a traditional publisher. Here we had a team of 5, all with Sun’s, all connected to the internet. The original architect had tried to create a resilient design, but didn’t know what he was doing. It was over-engineered and full of single points of failure. I simplified it… and then started to automate tasks; want to register a new DNS domain? Run this command, answer the questions; it will create the DNS entries on primary and secondary, create the email form, PGP sign it, send it to Internic/Nominet/whoever. Want to add a new dialup? Run that; it’s create forward/reverse DNS, tacacs, mailboxes, etc etc. New web site? Cisco IOS config…

When I moved into the central “systems” team I started to automate their processes. I even got zsh working on Windows NT so I could automate tasks the help desk commonly did. Automated monitoring (“Big Brother”, for those who remember that). Automated paging (Kermit, talking to a TAP service).

My third job was in a megacorp; I now had a thousand servers; more than the previous company had people! My first task was in Y2K readiness; how can we test these thousand servers are all working properly so the Y2K command center can tick their box.

I created a tool that could check all the servers, centrally coordinated (via ssh) with tunable (and self-tuning) parallelism. It would run in under 30 minutes. I wasn’t on call for Y2K, but the 2 people who were basically just watched the fireworks from the Embankment (the office was right on the banks of the Thames) while the code ran. (The Windows and Novell teams weren’t so lucky; they had to work).

A good sysadmin is a lazy sysadmin; they automate everything. I was a VERY lazy sysadmin.

So lazy that the code remained in daily use for years afterwards, as a general purpose healthcheck. Because I was in operations I wrote code to make my life easier, and as a result every ops guy in my team gained the benefit.

Where it started to break down

The company wanted me to do more and more complicated work. Stuff that required focus. Systems architecture; engineering; deployment. Stuff that I couldn’t do if I was paged out at 3am because a server needed attention. I wasn’t the only one. They split the team into engineering and operations. Naturally the smart coders got moved into engineering; the remaining ops guys were more “press the button” types. Not stupid, by a long stretch; they could do some really detailed problem analysis and solve problems… they were good SAs. But they were more incident management focused, rather than problem management. They might document a solution and then next time it happened another team member could follow the document.

At the same time the engineering team started to be divorced from the daily operations. We were focused on higher level tasks set by management, rather than scripting and automating BAU activities. Barriers between the teams were created by this organizational distinction; the automators didn’t understand the ops problems, so they started to need requirements docs and ops wanted handover docs and the slow decline into your structured support model (level 1,2,3 even 4) was inevitable.

The ops teams were considered by management to be a “cog” and each person replaceable. This lead to “cheapest person” type policies, which naturally leads to off-shoring and stagnation; if a person in L2 learns enough then they move into L3, and L2 gains a newbie with no institutional knowledge.

Where the original idea to split engineering from ops was meant to speed up delivery (focused teams) the leaching of talent from ops lead to slower processes as formal handover between teams became necessary; just to protect themselves the ops teams required good tested procedures from engineering; waterfall enforced by organizational and managerial boundaries.

So what about SRE?

If you look back at my history, SRE is kinda what I was doing 20 years ago. Find a problem, automate the solution so it never becomes a problem again. Find a task, automate it. Find a multi-step process, automate it to one step.

This is why I say that SRE isn’t anything new.

But there is a change… and it’s a good one. It’s formalization.

When I did my ops work I was working mostly in a vacuum. I was a single contributor, inventing technologies and solutions as I went along. I worked with smart people who were also single contributors in their area. Sometimes the areas overlapped and, if you were lucky, the solutions could be combined. But mostly each person just ran their own way.

SRE, in comparison, is taking a modern software design approach towards the solution. Instead of individual contributors we now have a team approach. There may be agile techniques, but I don’t think that’s necessarily so important.

Today tools and technologies exist that were not around 20 years ago; JIRA queues, wikis, deployment frameworks (ansible, cfengine, salt,…). We can create a common technology stack within the team that anyone in the team can use. We no longer have “Stephen’s magic” and “Fred’s magic’ and “Matt’s magic”; we have automation.

This is a massive step forward.

So why will the SRE model work?

If we look at my history and look at SRE today, these common tools aren’t enough to overcome the “I was paged at 3am; I can’t code today” problem.

If SRE was just a “modern technique to an old solution” then it’ll die out for the same reasons that my team got split in 2000.

Fortunately, SRE isn’t the only change. There’s also the DevOps model. Now that has it’s own problems (a blog post for another day!) but it can mean that the layer that SRE is working on is the engineering team problems of the last 20 years.

When we look at containers and automated application delivery; when we look at the “cattle” model of software delivery; when we look at hands off automation and configuration management… the old school “ops” model is a lot smaller. Some of the work has been pushed onto the Dev teams; some of the work just disappears.

And this is why, in my opinion, that SRE will succeed; it’s not in a vacuum, but it lives within a larger operational model change.

Other challenges

Some regulatory environments (e.g. Payment Card Industry - PCI) have restrictions around dev/prod access.

Indeed PCI says

Reducing the number of personnel with access to the production environment and cardholder data minimizes risk and helps ensure that access is limited to those individuals with a business need to know.

The intent of this requirement is to separate development and test functions from production functions. For example, a developer may use an administrator-level account with elevated privileges in the development environment, and have a separate account with user-level access to the production environment.

In the SRE model, the operations engineer is a developer. So can SRE be adopted within these constraints? I, personally, think it can… but it requires additional controls around the process. Fortunately, the modern SRE tooling has a lot of process focus; after all, that’s what modern software delivery methodology is all about!

Conclusion

I was being slightly provocative in my opening statements. I do believe the the concept of SRE is nothing new. What has changed, though, is the practice.

Because of the mismanagement earlier discussed, “operations” has a bad name. The SRE branding is a way of avoiding this tainted reputation, but it’s more than just a re-branding. It’s a shift back to the lazy sysadmin role, and you really want lazy sysadmins because they can do things at scale that can’t be done otherwise.

This, along with the role change causing a responsibility shift, means that the SRE model is a potentially viable one for organizations to adopt.

Encumbering New Technology

Sun, 29 Apr 2018 17:36:42 -0400

One of the exciting parts of the “new world” of cloud is the ability to green field solutions. We don’t have the legacy requirements and so we’re free to do what we want.

Or so the evangelists would have you believe.

The past lingers on

The reality is that many people are closer to a brown field environment. The organisation their team is embedded into has a tonne of reporting (“is your machine patched?”, “who has access?”). These requirements came from years of systems mismanagement (the best appdev team in the world isn’t necessarily the best at running servers) and auditor questions. To provide adequate reporting a series of centralised reporting tools have been created to help prove to auditors that the environment is controlled; “we have 30,000 servers; 357 of them are running Struts; 3 of them are unpatched and have change records for next week” is a great story to be able to tell, but it requires a level of conformity to get there.

And now you come along with your dynamically scaling app running inside an environment that builds/destroys servers at will. You can’t even answer the question “what servers are your apps running on?” because the question is meaningless in your world; it changes from day to day, even hour to hour.

So new builds may start being encumbered with requirements that trickle down from the traditional compute environment. Of course the CMDB needs to be kept up to date; of course you need privileged access monitoring; of course you need configuration management tools; of course…

We need to prevent this encumbrance from crippling the benefits of the new world.

In some cases we might be able to kludge a solution. “If we give you a nightly update into CMDB of all our AWS EC2 instances, will you be happy?“. Beware, though; once a machine is listed in CMDB then it’ll be expected to have all the other corporate tools (monitoring, access management, backups…) installed. Be aware of the consequences of the kludge, and try to avoid it.

The harder, but ultimately (IMHO) better path is to go back to square one and design new controls and procedures that are better optimised for this design.

The corporate policies and standards are meant to be the distilled guidance of everything that’s needed, based on legal reading of regulations, from history of audit questions and issues, from experience. These become your go-to documents. You may also want to go to the source, because regulations can change without your standards. So read the PCI docs, read MAS, read HIPAA…

And then design solutions based around the requirements, not around the existing practices.

This is hard work on a political level and requires a fair amount of senior buy-in. It requires vision and leadership because the organisation is now taking a fair risk; will you build a good solution that will pass audit? Will you expose the company to data theft? Will the solution scale to multiple appdev teams (you’re not a unique snowflake; other teams want to do this as well).

If you have management that support you, and can demonstrate compliance to corporate and regulatory policies and standards then you may be greenfield enough and avoid too many encumbrances.

New encumbrances

Because this is new and because you’ve now designed new processes and procedures, the people reviewing your work will be looking for edge cases. How can this new thing go wrong? It’s a natural consequence of trying to protect the company, the data.

The problem is when a potential gap is found. Is this gap already present in the traditional compute space? Is it known? Has the company decided this is OK? If so, it may be a problem you don’t need to fix; acknowledge the gap and move on. Don’t encumber your work with things that aren’t necessary.

Imagine building a PaaS solution and being required to tell the application operate team everytime you are going to reboot a VM? Technically it’s possible… but it’s stupid, and negates some of the benefits of a PaaS.

We see this in various places, especially in new fields of automation; for example, self-driving cars. I’ve seen people ask “once we get to fully automated cars capable of taking voice commands from the driver, how do we make sure the driver has a license?“. This is a perfect example of “new technological requirement” based on an existing requirement (must have a license, must be insured) but isn’t currently enforceable. People get excited by “we’re building something new; we can add this feature”.

The idea may seem reasonable on the face of it, and it may be a solution that is technically feasible. But this is a case of scope creep and can drag down your project.

“Just because you can, doesn’t mean you should”.

Don’t encumber your new solution with requirements that go above and beyond the problem you’re trying to solve. Try to keep project scope under control.

Conclusion

Very few projects are really greenfield. There’s always some small amount of legacy that you need to keep up with. Much of that legacy is there for a good reason, and you can’t just ignore it with impunity. However you can replace it with better processes and procedures, with better solutions. Sometimes going back to the drawing board and understanding the rationale is important.

Don’t just say “we’re doing DevOps” and ignore everything that came before; that’ll get you into trouble. However “we’re doing DevOps with these controls, procedures, enforcements, auditing…“; now you’ve got a chance to build things without the encumberance of traditional procedures.

When does an AI become alive?

Sun, 01 Apr 2018 14:33:49 -0400

This blog post is gonna be a little different; it’s more philosophical than most of what I write.

It rose out a question a friend asked:

“When does a biological AI become life?”

My friend asked me this because he felt that SciFi must have covered this topic, and I’ve read and watched more than my fair share :-)

Remove the limitations

Now, I found the restriction to “biological AI” is unnecessary limiting. I can understand the thinking (we’re biological, we’re alive), but the question can apply to a larger scope.

Let’s go larger.

I can see two possible interpretations of the question:

when does something become “alive” (in the same aspect that a rat may be alive, or a tree, or maybe even a microbe). We don’t need it to be an “AI” for this
when does it become sentient; we don’t need “biological” for this.

A related question is “what does it mean to be human”. Having just rewatched “Bladerunner” and “Bladerunner 2049” this question naturally came to me; it’s one of the key themes in both of these movies.

The problem with these questions is that we don’t really have good definitions of what any of this means; philosophers have been debating it for millennia.

Star Trek TNG asked this question a few times; e.g.

is Data alive (The Measure of a Man)?
Were the exocomps a life form (The Quality Of Life)?
“other forms of life” (ugly bags of mostly water)

Even the original series covered some of this (The Devil in the Dark). Star Trek Voyager took the concept even further with debate on whether a hologram could be sentient and if so, alive.

Slaves

A lot of these questions all come back to “creating an underclass” - ie creating a slave race. We believe enslaving people is evil, but if we create something or grow something and declare it “not human” then can’t we treat them like slaves? Asimov addressed some of that in his Robot stories as well.

So determining the answer to these questions (“what is life”, “what is sentience”, “what is human”) and determining how we should treat the entities we create may become a critical question that could have ramifications through history.

Will we remain human?

This may also have consequences for our own evolution (cf X-Men stories for mutants) or “Post-Human” (or Trans-human) for technology-enhanced humans.

What if we learn to create nanotechnology that can rewrite DNA to enhance muscle power; heart function; brain power? What if we have computer technology that can be implanted into the brain and act as an enhancement to memory, thinking, communication?

Will we define these people as “human” or will they be “sub-human”?

Since many of these technologies may come out of the defence industry and initially applied to soldiers who sign away their rights some of the early decisions may be against calling these people human, which could have long term ramifications. This was a thread in “Nexus” trilogy of books, by Ramez Naam. (Aside: I could do with some technology to help me remember names like that; I forgot the author and book title, but I knew exactly where it was on my book shelves, so had to go and physically find the book!)

My grandfather’s axe

Law loves bright lines; “This is human”; “this is not human”. It’s really not so good with blurred areas. This became a big deal in an obscenity case that appeared in front of the US Supreme Court:

I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description [“hard-core pornography”], and perhaps I could never succeed in intelligibly doing so. But I know it when I see it, and the motion picture involved in this case is not that.

We’re already starting down this path. We can replace limbs with artificial equivalents, we can replace organs (eg artificial hearts), we can even use organs from other species (pig hearts). So where does “human” end? When does man become Cyberman

So far the line appears to be “brain”… but what when we start to enhance that?

Brains!

For a long time we’ve had the ability to alter consciousness. “Beer, you’re my best friend! hic”. We’ve gone some step towards overcoming natural limitations (e.g. drugs to counter ADHD, reduce epilepsy). But what happens when we surgically make permanent changes to the brain?

Well, we’re already doing some of that, as well; cochlear ear implants where the implant connects directly to the nervous system. The are experimental artificial eyes to allow blind people to see. There’s even one person with an artificial eye that translates colours into sound.

So far we’re at the early stages of direct brain manipulation, but we’re definitely going down this path!

Human life

We can’t even tell when human life starts. When does a random collection of cells (egg+sperm) become “human”. Anti-abortionists claim “life begins at conception” but that’s meaningless and would result in a crime investigation being warranted whenever a mother spontaneously aborts (and in the first few days of implantation that’s common).

If we can’t even define this for humans, how can we expect to define it for our creations?

What even is Artificial Intelligence?

When I was a kid I thought that we would encounter alien intelligence… but it wouldn’t be from outer space; it’d be something we created. An AI, basically. After all, there’s no reason that an AI would be an artificial human intelligence (AHI); given the lack of sensory input and control mechanisms that human’s grow up with (your body!) there’s little reason to assume that an emergent AI would be human.

So we have another problem; how do we identify this intelligence? Clearly the Turing test is aimed at AHI, and not alien intelligence. Do we even have a way of identifying this?

Summary

We’ve come a long way from the original question (“When does a biological AI become life?”). It’s exposed a tonne of questions to which we don’t, yet, have answers.

But these are questions we do need to start thinking about; not from an AI perspective, but from a human perspective. When does man start? Where does he end?

The reality is that there is no hard line between “not alive” and “alive”, as exposed by the “abortion” question. The law tries its best, but there are legal, ethical, and religious components to the question.

When does an AI become alive? I don’t know!

And then there’s Roko’s Basilisk to be aware of…

How I learned to stop worrying and love the cloud

Mon, 19 Feb 2018 19:38:07 -0500

I’ve spent the past far-too-many years working in the finance industry, in mega-banks and card processors. These companies are traditionally very worried about information security. It’s not to say they always do it well (everyone makes a mistake), but it leads to a conservative attitude.

These types of companies end up creating a massive set of standards and procedures to protect themselves. “Thou MUST do this. Thou MUST do that.” Any deviance from the norm (in a weaker situation) is a risk and needs documenting and tracking.

There’s a number of assumptions involved in these standards. For example, the physical disk is in our datacenter; the SAs with logical access have gone through corporate vetting; the physical access controls to datacenters are known; network connectivity is known. At a higher layer, all internet facing services have been signed off (firewalls need to be opened to allow incoming connections); 3 tier architectures are defined and maintained; data loss prevention tools exist. In essence we’ve built both physical and logical barriers to control ingress and egress of data. You might even hear phrases such as “within the company’s four walls” to refer to this.

Now this scenario does assume staff can be trusted; there’s a whole another set of controls and processes to protect against that, but that’s kinda out of scope of this blog posting.

How does the cloud change this?

Before we can answer this question we need to consider what is meant by cloud. NIST defines cloud in terms of three primary services; IaaS (Infrastructure as a Service), PaaS (Platform as..) and SaaS (Software…). These really are rough demarcation points because a cloud engagement may include components of all of these (e.g. a Big Data offering is a SaaS, but it may offer “compute VMs” where you can install your own software… an IaaS).

I’m not going to consider SaaS here because, in essence, this isn’t really much different to traditional service outsourcing. HR departments have outsourced to specialist providers for decades; we’re now seeing this at the IT technology layer. You put your data into someone else’s software, run on their machines, and trust it. Where things have changed is that your outsourced provider now may, themselves, run on an IaaS. To trust the SaaS provider you may need to know their dependencies and the underlying IaaS provider.

Similarly with a PaaS; now you throw your software into someone else’s compute environment… which may run on an IaaS.

In both the SaaS and PaaS model we have to worry about controls at the service layer level, and potentially controls at the IaaS layer.

Which leads us to IaaS solutions. The worry, here, is commonly around the management of the physical infrastructure.

Sure, there’s a challenge around using Cloud Service Providers (CSPs) such as Amazon, simply because they provide so many services and tools. But we can work on that. We can define standards. We can build logical constructs that compare to the traditional four-wall security model.

But we’ve still got the lower layers. This is what Amazon called the Shared Responsibility Model.

Effectively, the CSP has replaced your Datacenter Operate team and portion of your SA team. And these are people you don’t know; they’ve not been through your corporate vetting process, had their fingerprint taken, peed in a cup for drug testing… how can you trust these guys?

The shared responsibility model also applies for PaaS and SaaS offerings; just the line of responsibility moves upwards. I like to call this “above the line” and “below the line”. The CSP is responsible for stuff below the line; you’re responsible for stuff above the line. Where that line is drawn is unique to each engagement and CSP.

Mitigations

The first thing we do is try to limit our exposure. Encrypt things where you can. Replicate as far as is required your existing controls (it’ll be a new implementation but you need firewalls, DLP, logging…). Determine what those controls really are, rather than just the decade old “this is how we’ve always done it”.

There are some cloud specific new stuff; always encrypt data at rest (block store, S3, databases). Automate detection of misconfiguration (an open S3 bucket? That’s not right! Open egress from your VM to the internet? Huh, that bypasses DLP!). Can your application work purely on encrypted data, and never need to see the plain text?

But at the end of the day we need to have a level of trust in the CSP. What’s to stop them snooping memory from the hypervisor? How far do we trust them to run the “below the line” stuff properly?

What can we about that?

Trust

Here’s where things start to get a little fuzzy. We no longer have direct visibility into these operations. It’s all below the line, and so an opaque box.

But, really, this isn’t much different to any other technology outsourcing contract. The more I’ve delved into “cloud” the more I’ve realised that we already have (or should have!) processes in place to handle this. Far too frequently I hear people discuss the complexities of a solution in the cloud and I ask “How does cloud change this?”. Most of the time it doesn’t.

So what does cloud change? Staffing is an obvious one; people have access to physical machines that didn’t go through your vetting process. Access management is another; these staff don’t authenticate via your corporate systems. Patch management? Inventory management? Physical security? The list of things ‘below the line’ goes on and on.

Which is where we have to start to trust, but have a level of verification. There are now a number of certifications that companies can attain to help you gain confidence in the opaque box. The primary one is the SOC2 report, which is part of SSAE18 and replaces the older SAS70 reports.

The SOC reports can be used as a baseline and audited by a trusted auditor. You don’t want a SOC report from “Joe’s bait and audit shop”! A properly performed audit can help provide assurance that many of the below the line controls a company would provide on-premise are also being met by the CSP in the cloud. You may also be interested in the SOC1 and SOC3 reports; all in all they provide a pretty comprehensive picture of the control state of a CSP.

Similarly, a CSP that claims to be Payment Card Industry (PCI) compliant will have additional audits via a QSA. It’s important to note the limitations of these certifications, though; a CSP like Amazon have over a hundred services but not all of them are PCI certified.

Summary

Personally, I’ve been using virtual machines from providers for over a decade (my linode was first deployed in 2004). But I’m not processing credit cards or bank accounts so I’m not a big risk. As long as I avoid being an idiot then my biggest opponent is a script; who wants to break into me? (No no no; that’s not an invitation!!)

But a bank? Ah, that’s a different matter entirely. They have data people want; they’re an attraction. Delegating control of some of the base layers is scary.

It all boils down to how much you can trust your CSP, and how much you can verify that trust. SOC reports, COAs, contractual obligations and insurance can all help with this verification and indemnity.

10 years ago I would not have placed critical processing in the cloud but, today, I think things have matured enough that PCI data can be handled there securely.

If you look at most of the recent “cloud based” data breaches they’ve not been at the CSP layer of the shared responsibility model, it’s always been at the customer layer; most frequently an open S3 bucket, but also open unauthenticated mongoDBs, and similar.

Yes, there is a risk your CSP might be broken into… but this is a micro-hole of a risk compared to the tank-sized hole you’re unwittingly building!

Manage the risk accordingly.

The cloud is not your friend

Sun, 04 Feb 2018 19:45:10 -0500

Even after all this time I hear statements like “Oh, we can just run our code in the cloud”. This is the core of the lift and shift school of cloud usage.

And these people are perfectly correct; they can just run their stuff in the cloud. But it won’t work so well.

I’ve previously written about lift and shift issues, but here I want to focus on the “resiliency” issue. People get annoyed when I point out that their design is unreliable and subject to failure modes.

“Oh but the cloud is reliable!” they cry. “Major companies like Netflix and even Amazon themselves never go down.”

If only it was that simple…

Existing designs

In a traditional datacenter environment a lot of work has been put in place to make underlying infrastructure reliable. We spend lots of money trying to get to that mythical “five nines” reliability. We build clusters of servers and use VMware vmotion to migrate workloads around the cluster to allow for nodes to be taken down for management. We dual connect machines to independent network switches. Similarly for SAN fabrics. And, on top of all that, we then duplicate all this in another datacenter and configure our application stacks for HA (e.g. Oracle data replication, global load balancing).

This is expensive to build, and hard to maintain. And very easy to get wrong. Who here has not been stuck on a “DR Failover” call where the DR component hasn’t started up properly because testing has assumed clean shutdown of primary… something that doesn’t happen in a real failure.

No wonder people want to move to the cloud; it’s so much more reliable! Except…

The cloud is not reliable

If you look at SLAs from major Cloud Service Providers (CSPs) then you’ll see they talk about a service SLA. Let’s take Amazon EC2 as an example:

The Service Commitment does not apply […] that result from failures of individual instances or volumes not attributable to Region Unavailability

Basically, Amazon provide a “four nines” SLA for a region, and not for individual running instances.

Your CSP running instance is probably less reliable than the VM you run in house. Maintenance likely isn’t going to happen on your schedule!

Plan for failure

This is a core requirement for moving to the cloud. If you want to get cloudy infrastructure to be as resilient as your on-premise stuff then you need to build solutions that match the existing patterns and designs. You may no longer have the ability to do things like VMotion, so how will you take instances down for patching? You will need to replicate databases, even cross region. You’ll still have requirements for DR testing, and this will be even more important, because of the lower underlying resiliency.

So now you have more VMs, more data transit charges, more headaches… this cloud lark isn’t looking so friendly now, is it?

Or you can look at how companies like Netflix actually get their reliability; their application assumes failure.

This turns the existing reliability model on its head; instead of having four-nines infrastructure that the app can rely on (and so mostly ignore), assume we have three-nines (or even two-nines!) and the application has to make up the difference.

Developers now take on the majority of the resiliency requirements. There’s a whole new set of programming patterns that need to be used; keeping state externalised, using a circuit break pattern, providing service state visibility… Tooling such as Netflix Hysterix can help with this.

Both approaches are possible; each have their pros and cons, challenges and costs.

Conclusion

You can make friends with the cloud. It requires thinking about. An application redesign host a lot of up-front costs but can lead to long term benefits. A lift-and-shift approach is simpler but can bring in a lot of legacy processes and procedures (“ugh, another broken DR failover!”).

Whatever you do, though, you can’t ignore resiliency and reliability in the cloud; you don’t get it “out of the box”. Done properly, however, then the cloud can start to work for you, and you’ll be best of friends and never want to work with traditional methods ever again!

What we can learn from the rebellion leadership failures in The Last Jedi

Mon, 22 Jan 2018 20:31:28 -0500

This is an odd post for me. I’m terrible as a manager. I’m terrible as a team leader. I think I’m good as a teacher and mentor, but that’s a different role. Lead by example, teach what I know, learn when I can. I’ve definitely not been in the military. And yet I’m about to write about effective leadership… or maybe bad leadership.

Finally I get to see The Last Jedi. And one thing stood out.

(“One thing?” Hush from the peanut gallery!)

Poe make a decision; he ignores his command hierarchy; he achieves his goal, but at terrible expense. As a result Leia demotes him. Then, and this is the part I have trouble with, Holdo, cuts him out of the command loop and tells him to shut up and obey orders.

I’ve heard that being a good soldier means “obey orders”. I’ve also heard that if you go beyond being a grunt then you’re meant to be able to think; an officer is meant to be able to make decisions in order to best obtain the objective. Even soldiers aren’t automatons, and the higher up the chain you are the bigger the perspective you get. The officer on the ground needs to be able to evaluate the local tactical situation in terms of the greater goals.

This is true in a corporate environment. Well, assuming senior management know what they’re doing :-)

As a lead engineer you’re meant to know the organizational goals and the immediate product goals, and make decisions (design, architecture, engineering) that best achieve those goals. If you have questions then you’re meant to engage your leadership team and discuss it with them.

Same if you’re a program manager. Or an SME. Or anyone! By understanding the goals, you can deliver better.

In an organisation, the worst thing your leadership can do is tell you “just do what you are told”.

As a leader you want your team to be invested in your projects and plans, and that means explaining the rationale to them. People work better when they understand things. Even better, they may not inadvertently work against the plan.

I’ve been in the situation where my management tree got replaced; basically they all quit and new people from outside the company came in. They brought their own vision and plans. However they were unable to justify it. It boiled down to “I’m going to throw away a working solution and replace it with a new technology because that’s what I want”. There was no logic, no rationale; indeed the replacement solution would be less secure and performant than the existing one. At least from my perspective as the SME in the area, and management did not explain otherwise (and 5 years later I still believe I was right).

In my case I found another role in the company so I didn’t have to fight that problem.

In Poe’s case he felt he had to take direct action (took control of the bridge) and potentially compromised the master plan. A whole mess could have been circumvented if Holdo has just spent 1 minute explaining the strategy to a person who had previously been part of the trusted advisory council (as much as the rebellion had any structure).

Summary

Good leadership requires all levels of the management tree to be to explain themselves. Simplistically, the seniors set the direction, the middle management set the targets, the juniors set the tasks. But at each level management must listen to the layers below; they must hear the questions, hear the problems, hear the ideas. They must be able to integrate new information from SMEs and respond beyond “because I say so”.

If you can’t explain, you’ll end up with people, in good faith, working against the plan.

Can't Patch, Won't Patch

Mon, 15 Jan 2018 15:09:21 -0500

Whenever a new “critical” vulnerability is found, the cry goes out across the land;

Patch!
Patch!
Patch!

Whenever a major incident is caused by known vulnerabilities the question is always

Why didn’t they patch?
We’ve known about this for months!
They should have patched!

Sometimes this is valid criticism, and learning why the organisation wasn’t patched can lead to some insights into failure modes.

Other times, however, it may not be possible.

Why can’t we always patch?

Sometimes patching raises challenges that aren’t always obvious. An annoying use case is where the computer may control some regulated piece of equipment. Let’s take the X-Ray machine at your dentist’s office; I’ve seen some of those controlled with software running on Microsoft Windows. I don’t know about you, but I’d rather not have non-experts apply a Windows patch that causes radiation exposure 100x greater than planned! Further, in the medical field, may be computers controlling CAT or MRI machines, which cost megabucks to replace. Now you may argue that users shouldn’t buy into this sort of hardware in the first place, but when it’s that industry standard then they don’t have a lot of choice. Here we’re definitely into “can’t patch” territory.

Related to this are the “appliance” type devices, where you are forced to wait for a vendor to release updates. You can’t patch until that happens!

Another scenario may be that legacy hardware needs to be supported; the software that drives that hardware doesn’t work under newer versions of the OS, so you’ve been limited to old unsupported unpatchable installs. In some cases there may be a path, but the cost is prohibitive; in other cases it may be a legal requirement (document retention requirements are big here; a need to restore backups from 7 years ago means needing to maintain equipment that can drive the hardware to do the restore). We may be in a combination of can’t/won’t patch.

And, of course, there’s a potential resource constraint issue; how long would it take you to patch your estate?

Whatever the reason, sometimes patching just isn’t feasible. Pointing fingers at victims of a cyber breach and shouting “you shoulda patched!” without knowing the full set of constraints the organisation was under doesn’t help you learn from these incidents and prevent them in your own organisation.

Compensating controls

So if we can’t always patch, then what can we do? This is where the “defense in depth” type controls can come in handy.

Web servers should be behind a WAF, for example. Can’t patch against shellshock (to take an example) because you have a gazillion instances? If your WAF can block the pattern, then it should do so. Beware false positives, of course…

Legacy devices may need to be placed on a protected network behind firewalls, requiring explicit permission (jumphosts, firewall ingress rules, whatever) to be reachable from the core network. Of course, such devices may need to be protected from each other, leading to a proliferation of firewall rules!

Emulation may work in some cases; run the older app under an emulation layer. It may work and may mitigate some of the issues…

There’s no one solution to these problems; the mitigation will be unique to the challenge.

Risk management

In some cases no mitigation may be possible (e.g. an MRI scanner that expects to write results to a standard Windows share; WannaCry will happily encrypt all that data!). So now we need to look at managing the risk.

The first step, as always, is to be aware that you have a risk. Inventory tracking, patch state… all the standard vulnerability management stuff is a prerequisite; you can’t evaluate what you don’t know.

Perform an assessment; is an unpatched device really at risk? My TiVo runs Linux at its heart; is it at risk to Meltdown or Spectre? Given the software stack on top, I’d guestimate “low risk” (even if the hardware was susceptible, which it may not be) due to the limited input channels to the OS. Is your Windows 95 machine, running your door swipe system, at risk?

Determine what mitigation is possible. Stop the caretaker using the door entry PC to surf the web :-)

And so on. Y’know… all the standard risk management stuff.

At the end of the day you end up with an evaluation of the criticality of the exposure, the likelyhood of it happening, the consequences. Outside of a VM farm (including clouds), is Meltdown a big risk to your organisation? Possibly not. Shellshock may have been, though.

And then you decide whether to carry that risk, or plan a remediation strategy (spend money, upgrade, convert backups to a new format…).

And perhaps consider insurance, just in case you’re wrong! (Although can you insure against reputational loss when you leak a gazillion credit cards?)

Summary

Not every issue can be patched. Not ever issue needs to be patched. But organisations need to be aware of their risk and determine an approach.

Risk management and vulnerability management needs to be a critical function in any organisation. Doing it wrong can cost lots of money (either in wasted effort due to emergency ‘patch now patch now’ mentality, or in incident response after a breach). It’s not a side issue to be minimally funded just because some NIST or SANS chart tells you it should be there.

Meltdown and Spectre

Thu, 04 Jan 2018 18:57:41 -0500

Unless you’ve been living under a rock, you may have heard of two panic panic panic bugs, known as Meltdown and Spectre. People are panicking about them because they are CPU level issues that may impact almost every modern CPU around. Meltdown is Intel specific, but Spectre affects Intel, AMD, and potentially others (Redhat claims POWER and zSeries is impacted).

What is the problem?

In short, modern CPUs may execute instructions out of order, especially when the order doesn’t matter. If you have

a:=1
b:=1

then does it matter what way round they run?

Now modern CPUs also speculate about branch instructions and may pre-emptively execute the branch it predicts as being the one to be followed. If the guess about the branch is correct then the out of order execution results are valid and your code runs faster. If it guesses wrong then “no harm, no foul” we just throw away the guessed results and run the right branch.

Except it turns out there is a foul. The predicted branch impacts the CPU cache. And it turns out it’s possible to use this fact to derive further information about memory locations you shouldn’t normally have access to.

The result is an information leak. Inside a single VM one process could read kernel memory data. Want to read secrets out of memory? It’s theoretically doable. Worse, on a virtual server one VM could read memory from another VM. Oh dear, do you trust everyone sharing the Amazon Cloud server that your EC2 VM is running on?

Why are people panicking?

CPUs have bugs all the time. Most of them can be patched at boot time with a microcode update. But in this case the problem is at a lower level; it’s all to do with the cache. Now the Meltdown issue can be mitigated at the OS layer by modifying how the kernel handles virtual memory. This mitigation isn’t free; it may have an estimated 5->30% performance impact. That sounds like a lot! The 30% impact, though, is almost the worst case scenario (in testing it required a specific pattern of reads to an SSD). More commonly the impact is 5%, and if your application is compute intensive then there may be almost no impact (I saw some gaming desktop testing where the patch had an impact smaller than measurable error). But 30% is the headline number.

The Spectre bug apparently can not be fixed through software. We can block known paths to exploit and make it harder to exploit (it’s already pretty hard to do more than a proof of concept) but we can’t block it totally.

So people are now panicking like mad; do they apply the patches ASAP to prevent exploitation and risk the performance hit, or do they leave servers exposed?

Why am I not panicking?

In essence this is a read-only local privilege escalation bug. Sure, it’s at the hardware layer rather than at the kernel, but the impact is still comparable to that of local privilege escalation.

In comparison, Shellshock was remotely executable and trivial to exploit. That’s scarey.

Many places tend to have a ‘one workload per VM’ model. Privilege escalation, there, isn’t so great a risk. However hypervisor technology means that one physical server can serve multiple logical VMs and so multiple workloads. Similarly container technology allows one VM to handle multiple workloads.

So I look at it from layers:

“Top priority”
Hypervisors should be patched. This protects one VM from another. A cloud service provider should treat this as critical, because they host untrusted workloads (indeed Amazon and Microsft have already patched and rebooted a majority of their servers; operations at scale!). Inside your enterprise your VMware/KVM/Xen/Openstack/… environments should be patched. Any host running containers (eg docker) should be patched. Any on-premise cloudy environment (e.g. Cloud Foundry, Apprenda) should be patched.
Basically any environment with “neighbours” should be treated as top priority priority, so one neighbour can not impact another.
“High”
Server OSes should be patched, of course, but the urgency is less.

Risk management

Of course you don’t want to leave known vulnerabilities in your environment, but the whole point of measured risk management is to understand the impact of a vulnerability on your organisation and prepare an appropriate response. Ask yourself questions around ease of exploitation, risk of exploitation, consequence of exploitation. At present this appears to be a local read-only privilege escalation issue; what is the impact?

Can remote attacks trigger the bug? Well with Spectre there’s a proof of concept javascript attack, so maybe ensure your desktop browsers are up to date. Do you have code that may dynamically build eBPF filters into the kernel? If so, ensure the inputs are trusted (the right thing to do, anyway).

Your organisation should already have vulnerability management processes to handle new vulnerabilities. Treat these the same way. It doesn’t matter that it’s a hardware issue; it’s just another issue. Follow your patch guidelines, follow your processes. Don’t panic.

Unless you’re a cloud vendor of course. Then panic and patch immediately! You don’t want one customer stealing data from another. Your risk analysis is different to a traditional enterprise!

Summary

Bugs are bugs; whether it’s a hardware bug or a kernel bug, or an application bug. Follow your processes.

What’s interesting is that this is next along the class of attacks at the hardware level. Rowhammer showed that memory could be made to do things outside of the knowledge of the OS. Meltdown and Spectre show that the CPU, itself, can be abused in a similar way. What will be next? What core assumption are we building on that is shakey? That might bring the whole house of cards known as “secure computing” tumbling down?

Douglas Charles Harris

Sat, 30 Dec 2017 23:13:02 -0500

My Dad died, at the age of 79, in Tenerife doing what he loved… falling asleep in the sun while on holiday.

I might write another post around what Dad meant to me, but I’m going to limit this to the ceremony itself.

The cremation was held on 13th December. Dad was a “believer” but low key about it (Mum doesn’t believe) so we had the local priest perform the ceremony. All in all, as these things go, it was OK.

The procession to the crematorium consisted of just the hearse and a car containing family (Mum, me, Jason, Stuart, Tori). The funeral director walked in front of the car while on our local back streets, and tears came to my eyes at the amount of deference he showed to the body whenever he passed it (paused, took off his hat, bowed his head). It’s probably the most repsect Dad has ever been shown! The drive was pretty slow and I remember laughing to myself; Dad would have been annoyed if he’d been stuck behind the procession with no way to overtake!

The ceremony itself was almost as we planned; we’d given the priest a good outline and set of jokes. For example, Mum had written that Dad always wanted to do stuff his way, and that way was normally the right way. I added a joke; “well, except for when he electrocuted himself changing a light bulb. Or when he fell off a ladder cutting a tree branch. Or when he broke the computer. Or…” The priest told it well, and a number of people mentioned that he’d known Dad well. Oh nononono, we just gave him a good story outline :-)

The music choices were:

Un bel di vendro - Madame Butterfly
Dad loved this song
My Way - Frank Sinatra
I think this shows up in many a funeral, but it also fit in with the joke
Fly Me To The Moon - Frank Sinatra
This played to another joke; Mum can’t stand Sinatra but she had two songs by him because she loved Dad so much. This song (or a variation from someone else) was the first song they danced to when they first met, so it was fitting that it also be their last song.

One further poem was read out during the service:

Remember Me

To the living, I am gone
To the sorrowful, I will never return
To the angry, I was cheated
But to the happy, I am at peace
And to the faithful, I have never left

I cannot speak, but I can listen
I cannot be seen, but I can be heard

So as you stand upon the shore
Gazing at the beautiful sea, remember me
As you look in awe at a mighty forest
And in its grand majesty, remember me

Remember me in your hearts,
In your thoughts, and the memories of the
Times we loved, the times we cried,
the battle we fought and the times we laughed

For if you always think of me,
I will never have gone.

Goodbye, Dad. Thank you for loving me, for believing in me, for everything you did for me.

Technology is not enough

Sun, 19 Nov 2017 15:33:31 -0500

“To summarise the summary of the summary; people are a problem” - Douglas Adams, The Restaurant At The End Of The Universe

The above quote is one of my favourite jokes (I’ve used it in a previous post); it highlights how people can complicate any situation. We can try to avoid this by automating as much as possible but, at the end of the day, there’s always a human involved somewhere; even if it’s the team that manages the automation!

A few weeks back I was down the pub with some friends and they got to talking about a Red Team run against the solution they managed. Now I was interested in this because I’d been involved in some of the design and implementation of this solution.

It had great technology in place; firewalls, segregated networks, jumphosts, keystroke logging, multi-factor requirements and, of course, automation at multiple levels of the stack.

This team had developed a robust set of processes and procedures. Key credentials were stored in a password vault that only non-operations staff could access, thus enforcing “two person” controls. All admin access was audited and tickets generated for review and signoff.

Now there was a gap; the product’s admin interface is accessed via the same server as applications running on the platform. This isn’t uncommon; in a PaaS many of the admin tools run on the PaaS itself. This made it hard to segregate admin API traffic from application traffic so this stuff was hard to block at layer 4, and layer 7 would have introduced complications and performance problems - especially for microservice architectures, adding overhead to every service call really hurt things.

So the team accepted this risk.

I’m sure you can see where this is going…

The Red Team were unable to break the design; the technology was secure. The endpoints had no flaws (no SQLi, no XSS, no buffer overflows). The interactive admin pathway was secure and they couldn’t break that.

So they turned to the staff. This path bore fruit. An intern in the Ops team followed a phishing link and entered his AD details. The Red Team used those to access his VDI. They found, on his desktop, a text file with the admin password in it. They were able to use this to access the API.

At this point the detective audit controls tripped but it was too late; an attacker had admin in the environment and it’s now considered compromised. The best we can do is limit the spread of damage and determine if any sensitive data had been exposed.

At this point it’s tempting to point fingers. Why did an intern have the admin password? Why did he store it on his desktop? Who was his supervisor? But that doesn’t help; finger pointing doesn’t solve the underlying problem. Stop, step back, take a minute, slow down.

The human weakness

Any technology deployment needs to recognise that, at some point, humans will be involved and that humans will make mistakes.

Many organisations have “cyber awareness training” processes. It’s even a requirement in some regulatory environments. But these tend to be a “one size fits all” training, and don’t take into account unique circumstances; an operations person with admin access to a PaaS has a different risk profile than an admin assistant and the consequences of succumbing to an attack is different, so why do they have the same training?

Additionally this training is typically done on a yearly basis with some form of online “click click click” course. It becomes more of a box ticking exercise than a true training exercise. I really loved learning about “Know Your Customer” requirements and “structuring money transfers”; that training was so relevant to my job!

Also our poor intern probably fell between training cycles and never did any of the courses; he hadn’t been there long enough! Even new hires typically have a window to get the training completed by, and in the mean time they may have access to production systems.

What can be done?

Earlier I described a number of technology controls this team had in place to protect their service. What was lacking was a set of non-technology controls. An obvious one might be “no one gets admin access to production systems before they’ve completed these training courses”. Another might be “people with privileged access to production systems require those additional courses”. Yet another might be having some people require refresher courses more frequently than the minimum standard.

Of course there are reasons why organisations don’t do this; primarily cost. If someone can’t do their job until they’ve done a handful of courses then that’s a week or two of time wasted. Who has that kind of slack? I would counter than these are false economies and may open you to greater expense in the future.

This means that the people creating courses also need to be trained, in order to create effective security programs. SANS calls this Securing The Human.

Conclusion

I’m not a security training expert. I don’t know the answer. I can create great technology controls, but if I don’t take into account human fallibilities then these controls will not be sufficient. I can teach people on a one-on-one basis (and I do!) but this training doesn’t easily propagate through the enterprise.

Organisations need to create a “secure human” process that encourages culture change so that new people automatically get inducted into the correct way of doing things, both via tollgates and via co-workers.

And this means more than pop up boxes “By accessing this system I promise not to be a bloody idiot [ OK ]“. They don’t work. But that’s a rant for another day.

The three tier network is dead

Sun, 05 Nov 2017 13:59:28 -0500

It’s a fairly common design in enterprise networks; a three tier network architecture, with firewalls between the tiers.

Typically these layers are split up with variations of the following names:

Presentation Layer (Web)
Application Layer (App)
Data (or storage) Layer (Data)

Typically you may have additional tooling in front of each layer; e.g a load balancer, a web application firewall, data loss protection tools, intrusion detection tools, database activity monitoring…

This gives a relatively good set of protections; attackers from the internet can only see servers in the DMZ; if this server does get attacked then the intruder can not see the data layer; they can only see the application server.

It’s not perfect, of course. Many organisations don’t have much protection between servers inside each tier; an attacker who breaks into one web server may be able to attack a second web server and use this to pivot through the organisation. “Micro segmentation” is a concept that could protect against this, but it’s very difficult to retro-fit this into an existing environment.

Modern application design

Unfortunately this type of environment doesn’t necessarily work so well with modern dynamic applications. If you deploy something like CloudFoundry then your presentation layer and your application layer may both be on the same network segment, or even running on the same server. With docker compose or swarm you can build a 3-tier design on a single machine, and you can have many 3-tier apps all segregated from each other on the one box. Dynamic scaling can mean that firewall rules may have to be updated programmatically. Network Address Translation (NAT) can make it difficult to write firewalls at all, since the firewall can no longer distinguish between the applications.

The new 3-tier network

The idea of the 3-tier network is a good one, and we should try and keep it. But the separation of network control devices (eg firewalls) and application servers causes issues. Fortunately technology has moved on and we can start to take advantage of this

Tier 1

Modern load balancers can do a lot more than just direct traffic. They can also inspect it, acting as a WAF. They can validate API traffic matching XML or JSON rules. They can act as authenticating endpoints, eg talking back to an LDAP server…

Pretty much a modern device like this can be the presentation layer. Start to remove traditional web servers out of the DMZ and replace them with these appliances. This is a good thing to do, anyway, because it reduces the footprint of devices exposed to the internet. Why run hundreds of full RedHat instances and JBoss servers with the associated management requirements when a handful of dedicated appliances can do the job?

Once we accept the appliance can do the presentation layer role then we’ve also removed one of the problems holding up the deployment of modern applications.

Tier 2 and 3

This is where things get a little more challenging, but where the concept of micro-segmentation comes in.

With Amazon Security Groups, with Docker Swarm networks, with Kubernetes network structures, with Cloud Foundry Application Security Groups… all of these technologies can control the egress of traffic from a container or server, and in some cases also control ingress rules. These rules allow you to define tightly controlled access paths; application A instances can only talk to other services and data stores that have been authorised. Since the rules are defined to the application it doesn’t matter where they are running; the rules follow the instance.

This is new and scary for traditional network architects, who like to know where traffic flows are. Paradoxically this lack of knowledge is what increases security; we don’t need to care about IP addresses any more; instead we define rules in terms of intent:

“I want application A web server to talk to API servers from application B and C”
“I want application B API servers to talk to application B data servers”
“I want application C API servers to talk to application C data servers”

Summary

This sort of approach is still its infancy in many places. It’s very hard to retro-fit this model onto existing networks; it requires a massive discovery process, especially when the majority of communication may be within a single tier and so no existing firewall rules may be present.

In new deployments, however, it can be made a requirement. Ensure your Amazon instances are behind locked down security groups that restrict egress as well as ingress. Monitor deployments and alert on overly open access (does your database server really need the ability to reach any server on the internet?). Set up flowlogs if you need traffic logging.

In a container world, use orchestration tools that can define communication between services.

We’re now moving from a “discovered” communication model to a “defined by intent” model.

The 3-tier network isn’t dead, after all… it just changed to a multi-tier micro-segmented network!

Software for my digital safe

Sat, 28 Oct 2017 21:58:54 -0400

A few weeks back I completed my Arduino hack for a digital safe.

What was missing, however, was the software to drive it.

One requirement I had was to let it work with password managers. I also had the idea that maybe remote access (e.g. control the safe while away from home, to grant a guest access) might be useful.

This kinda meant it’d be easiest to do as a web site, with internet connections forwarded via the router.

In case other people also want to use this software, I decided to write it in Go; now Linux, Windows, Mac and other users could run it on their machines.

So, here it is. That zip file is 7Mb in size and contains the source, plus binaries, and the web site.

Note that at least once Windows Defender false alerted on the windows binary. If you don’t trust my compilation then read the source and compile it yourself.

Source code

The zip file contains a number of files. The source code is in the safe.go, vend, vendor, build.sh files. You don’t need these to run the program. If you want to recompile it yourself then set GOPATH and run go build safe.go. The build.sh script does this for Linux/MacOS.

Running the code

What you do need is one of the executables, for your OS (eg safe.linux.amd64) and the static/ directory. This directory contains the web site contents, and a simple JPEG image file, which we’ll discuss later.

Running the code may be as simple as running the binary. With no additional configuration it will run the web server on port 5000 and try to find the safe on a default serial port (/dev/ttyUSB0 or COM1:).

Set up

The default values are probably not correct for you. You might see an error such as:

$ ./safe.linux.amd64 
Using configuration file /home/sweh/.safe.cfg
  Serial Port = /dev/ttyUSB0
  Listen Port = 5000
  HTML static = /tmp/safe/src/safe/static
  Lock image loaded

Could not open serial port: open /dev/ttyUSB0: no such file or directory

The first line lets you know what configuration file can be used to override the defaults. On Windows it will look for %HOMEDIR%%HOMEPATH% or %USERPROFILE. On Linux/MacOS it will look in $HOME. Examples may be:

C:\Documents and Settings\sweh\.safe.cfg
C:\Users\sweh\.safe.cfg
/Users/sweh/.safe.cfg
/home/sweh/.safe.cfg

This file is a JSON format file.

{
  "ListenPort":12345,
  "SerialPort":"COM3",
  "AuthUser":"username",
  "AuthPass":"password",
  "HTMLDir":"/path/to/static"
}

Each line is optional. The smallest file might be

{
  "SerialPort":"COM3"
}

If you have multiple lines then separate them with a ,

ListenPort	Defaults to 5000. What port to run the webserver on
SerialPort	Defaults to COM1 or /dev/ttyUSB0. Where the safe is
AuthUser	If set, require this username/password for the website
AuthPass	See AuthUser
HTMLDir	If the code can not find the static directory, you can set it here

If you plan on making this available over the internet then I strongly recommend setting AuthUser and AuthPass.

On Linux, if you have multiple USB serial ports then it may make sense to use the /dev/serial/by-id/... path to the serial port, rather then /dev/ttyUSB... values to ensure that the right serial port is picked up (just in case the order changes on next reboot).

Successful startup

Once you’ve correctly configured the software, the startup would look something like:

Using configuration file /home/sweh/.safe.cfg
  Serial Port = /dev/serial/by-id/usb-1a86_USB2.0-Ser_-if00-port0
  Listen Port = 12345
  HTML static = /tmp/safe/src/safe/static
  Lock image loaded
  Successfully connected to Safe
  OK Safe is unlocked

The “Successfully connected” means it was able to talk to the safe. The last line is the current state of the safe.

You can now access the website. In this case, I’m running on port 12345 and so http://localhost:12345 would work.

You can see that web site design isn’t my strong point. This could have been written by somebody back in 1996… and will probably still work in a browser from that time!

The controls are in the left frame, and the results on the right.

Status
Reports the status of the safe (eg unlocked, locked, etc)
Open Safe
If the safe is unlocked then it will open the solenoid for this period of time. On the right we’ll see a countdown. During this period the door can be physically unlocked and opened.
```
OK opening safe for 5 seconds
OK opening safe for 4 seconds
OK opening safe for 3 seconds
OK opening safe for 2 seconds
OK opening safe for 1 seconds
OK completed
```
If the safe is locked then you’ll get an error instead
Test password:
This will let you verify the password you enter is correct. It will not unlock the safe. This may be of use before a session is started, to verify the password you entered to lock the safe is correct
Unlock Once:
If the password is correct then you can press the “Open Safe” button once. Once the open period is over then the safe is locked again. This is very close to the original “enter code” function on the safe keypad. It may be used, for example, to grant temporary access to the safe remotely, or just to gain everyday access without clearing the password out.
Unlock Permanent:
If the password is correct then it is cleared out of the safe and the safe is returned to unlocked mode. Unlimited use of “Open Safe” is possible. This is useful when the safe no longer needs to be locked.
Lock:
As expected, if the safe is currently unlocked then a new lock code can be entered, and the safe locked. Any string up to 100 characters, consisting of letters (upper and lower case) and numbers is allowed.
Unlock file with image:
These options work the same way, but with the password embedded in a JPEG file; see the “Randomize” function for details
Randomize:
This will generate a random password. It then takes the static/lock_image.jpg file and embeds the password as a comment. It will then send the image to the browser; you should (all going well) be prompted to save the result.

Note: there is no encryption of the password inside the file; it’s just stored as a comment. You could look inside and file it. But it may be used to “hide” the password; eg inside a cat picture. No one looking at my_funny_cat.jpg would expect to see your safe password in the comments. It might stand out for someone using exiftool or similar, though!

If you wish, a different JPEG file can be used. The code should strip out unnecessary sections. Make sure you test carefully if you do this. Yes, your unlock picture could be an anime cat, if you really wanted it to be!

(Actually I don’t recommend using this option; it’s kinda just a joke function. You might want to remove it from the HTML so you can’t select it).

Normal password functionality

The safe door should be open and the bolts in the unlock position
The owner would enter a password (twice) and click on the “Lock” button
The owner can then test the password with the “Test password” button.
All looking good, the owner can then close and lock the safe.

Silly picture use case

The safe door should be open and the bolts in the unlock position
The owner will click the Randomize button, and save the image
The owner can then test this works with the “test image” function
All looking good, the owner can then close and lock the safe.
The owner should rename the image file to something innocuous.
The picture can be used to unlock the safe

Restarting

The safe can be powered down and powered up again. The PC software can be stopped and restarted. It should all automatically recover. If the serial port is disconnected while the software is running then you may need to kill and restart the software. But don’t worry; the password is stored safely inside the Arduino and the safe is kept locked :-)

Know your threats and defend accordingly

Sun, 22 Oct 2017 16:33:56 -0400

A couple of weeks ago I was asked a question around the disposal of SSDs. The question went along the lines of “In the old days we could just overwrite the disk many times (eg with DBAN). What should we do, now, with SSDs?”

Recently, a bunch of Infineon TPMs were found to have a flaw that generated weak RSA keys. This could have lots of impact, including Bitlocker disk encryption.

Around the same time, KRACK flaws were found in WPA2, the mechanism used to secure WiFi networks.

Earlier this year, flaws in how mobile networks work (“SS7”) allowed hackers to steal money from banks by stealing 2FA passcodes sent by SMS.

I can keep going on. But what do all these completely different things have to do with each other?

Their impact is different, depending on who you are.

Disk data destruction

Since that’s what got me thinking about this, let’s start here. How should you destroy the data on your SSDs? Now I could go into some depth on how the old methods didn’t really do a complete erase (eg due to bad block reallocation), and the wear leveling on SSDs makes it even harder. But this really missed the point.

Ask the question; who was going to do a low level forensics analysis of the disk? If you’re an average “John Smith” then the answer will be different than if you’re a bank storing credit card numbers.

Another question; is this my biggest risk? If you are using a laptop (or phone) or carrying about portable media (USB drives) then a bigger risk is losing the laptop, or having it stolen. This means you should really encrypt your data on these devices (eg with Bitlocker, on Windows).

If you’re a businessman with the laptop then you might also want to have a BIOS password to prevent Evil Maid attacks for industrial espionage.

So how do you clean a disk? If it’s encrypted, just destroy the encryption key! If you’re in a bank’s datacenter you might want to physically shred the disk. If you’re recycling your old home computer to give to your local charity then a full format (not quick format) followed by reinstalling the OS will probably be good enough.

There’s no one solution.

Infineon TPM

We can ask similar issues about the other issues; let’s take the Infineon one… this has some people worried because the TPM RSA key may be the whole root of trust for an environment. In particular, Bitlocker encryption may use this key (and we’ve just said that laptops should be encrypted!). So we should be panicking, right? Well, it’s estimated that it would take 1000 Amazon instances 17 days and cost $40,000 to break this.

Is your data worth $40k to an attacker? Or, rather, do they think it may be worth that much money? My personal laptop? Nope, not at all. A laptop used by SAs to login for support work? Probably not. A laptop holding personal banking details, or medical records, or mergers/acquisitions? That’s more likely.

As an individual, this issue doesn’t really impact me and I probably won’t update the firmware to fix this. As a security person for my company we risk-rated and created a priority fix.

KRACK

If you’re using Starbucks then you better be using VPNs or only visiting SSL protected sites, anyway, because MITM attacks are easily possible. KRACK doesn’t change this.

Your home network? Most people only use home WiFi to connect to the internet, eg for mail, browsing, streaming. This is normally SSL as well. Maybe you have a wireless printer? Or some unprotected “Internet of Things” device? In theory these may be vulnerable to KRACK… but who is going to attack you? What will they learn?

At work, though… many places have an employee WiFi network that’s behind the perimeter firewalls. Ah, now we have a risk that may need to be addressed. It may be that we should re-architect our WiFi designs and only allow VPN access from this network. Effectively treat it as being outside the perimeter and don’t trust it.

SS7 flaws in SMS

This one always causes fights. NIST has deprecated SMS as an authentication factor. So we should stop using it, right?

Well, once again, it depends. We all know passwords are terrible, and people pick bad passwords. Now the attack the average person cares about isn’t an attack on them, but an attack against the server; for example, an attack on the bank. This attack just randomly throws usernames and passwords at the server, and some may work! Our average person has their money taken. The defense is some form of 2FA, and even SMS is better than no 2FA.

If you work in the bank though, then your logins to the VPN gateway may well be targeted. For that SMS may not be good enough.

And if you’re the CEO of the bank worth a gazillion dollars then you may, personally, be targeted.

Conclusion

I just picked a random handful of attacks and risks, and demonstrated how these risks are different, depending on the scenario you are in.

As security professionals we have a habit of “absolutism”; if something is known to be weak then it’s considered bad. But this is a perfect example of “the best being the enemy of the good”.

For your average home user, a password manager is better than using the same password everywhere (heck, even a password book is better than using the same password everywhere!). SMS 2FA is better than just a standard password.

For a person at risk (e.g. vengeful ex; targeted by trolls; abusive partner; a high wealth individual; a famous name) the risks change; you may have your phone stolen, your laptop hacked, your email accounts targeted. Is someone willing to spend $40000 to decrypt your hard disk? If so, update your TPM and change the encryption keys!

An organisation (and the employees working at the organisation) have another set of challenges, and this may depend on the industry they’re in. We’ve heard stories of laptops being found in taxis with large databases on them.

We need to move away from the “one size fits all” mentality, and start looking at a “good enough” solution. Even “poor” 2FA is better than no 2FA.

Most people don’t need perfect security; they just need “good enough” security. If you’re in a risky situation then this XKCD may actually describe your biggest risk!

Adapting a digital safe to be computer controlled

Mon, 09 Oct 2017 21:58:54 -0400

For a number of years I had one of these cheap electronic safes. They allow for a combination to be set.

I bought this one from Harbor Freight in 2004:

This post isn’t about that safe though.

About 10 years later the safe started to stop working; the control panel stop responding, it made horrible noises… Fortunately anger lifting and dropping the safe got it working again to open the door. I pulled it apart and pretty much there’s just a motherboard and a solenoid and a ribbon cable to the front panel.

So I bought a new safe. I got this one from Staples

I was curious and took this one apart. And it looked VERY similar!

When I looked closer, the only difference between the two setups appeared to be the connector to the battery/reset switch had the alignment notch on the other side (the white connector in the picture). So I did a little kludging and put the new safe’s board into the old safe. And it worked! The old safe is still in use, with the new board.

So I had a safe with no electronics. I wondered if it was possible to replace the electronics with an Arduino, and set complicated passwords via a computer.

And that is what this post is about!

Requirements

An electronic safe. Pretty much most of these safes use the same technique as the two I have. We don’t care about the front panel or the electronics; all we need is the solenoid locking mechanism.
An Arduino Nano. You can find cheap clones on Ebay for a few dollars
An Arduino relay. The Solenoid needs more power than the Arduino can provide, so we use the Arduino to switch a relay, and this switches power to the safe. I found one for $4.
A DB9 female serial port plug. This is how we communicate to the safe.
A USB serial adapter. These are also on Ebay for a few bucks. Doesn’t need to be expensive! Indeed the cheaper ones are normally better; the real RS232 standard allows for voltages up to 12V, which is more than an Arduino can handle. But the cheaper devices tend to stick to USB voltage levels (5V) which still works and is compatible with the Arduino.
A 5V power adapter. I took an old USB adapter and cut the end off.

Overview of the solution

The Arduino will determine if the safe can be opened or not. It will be embedded inside the safe. To talk to the Arduino we use a serial port plugged into the PC. Using this we can send commands such as “lock the safe with password 12345”, “unlock the safe with password 12345”, “open the solenoid lock”. The Arduino will save the password into its EEPROM so even if the power is removed then applied later it will remember.

Originally I was going to use the USB port on the Arduino for both power and communication, until I realised that this would let me upload a new sketch and open the safe at any time. Doh! Fortunately there’s a software serial solution, so we don’t need access to the USB port.

So this is what we’re going to try and build (click for a larger image). Sorry for the awfulness of my sketch!

Putting it together

First remove the inside cover of the safe. This is normally just 4 screws.
Unplug the power/reset cable from the main board. Unplug the solenoid cable from the main board. Unplug the ribbon cable from the main board.
You can now unscrew the board from the safe door. We don’t need it any more.
The reset switch on the side of the door can be unscrewed and pushed through. This frees up a nice hole, which we’ll use later. If you cut the two wires to the battery compartment then you’ll be able to remove that cable harness as well. We don’t need that anymore, either.
At this point you should have the door cover free and the safe door has the ribbon cable and the wires to the solenoid.
We need to run a cable through a mounting hole on the back of the safe, and then through the hole in the door cover. We only need 4 strands of this cable (2 for power, 2 for serial). I used a few feet of spare ethernet cable I have. It works well. The unneeded 4 cables add strength and don’t hurt.
Now on the outside of the safe pick one of the twisted pairs (I used orange but it doesn’t matter). Connect them to the power supply. In my case I just cut the USB end off an old LG USB power adapter and wrapped the wires together. It’s important to remember which side is +5 and which side is GND, so I made ‘orange’ to be +5 and ‘white/orange’ GND.
Pick another pair (“blue”). These will go to pins 2 and 3 of the DB9 connector. You also need to connect pin 5 (GND) of the DB9 to GND of the power supply. I’m terrible at soldering so, for me, this was the hardest part! This is also why I chose ethernet cable; the thickness of the cable means it can be secured nicely to the DB9 and is unlikely to break free.
Take a break; we’ve finished on the outside of the safe. Now we’re starting on the inside
The two cables used for serial communication need to be connected to pins 10 and 11 of the Nano. Now I cheated a little, here. I took a standard cable used to connect to Arduino pins and cut one end off, and connected that to the ethernet cable. Now I can just slip the cable onto the pin. If you’re better at soldering than me then you could solder directly to the board.
The power cables need splitting; one set will go the Arduino (so another pair of standard cables), the other positive side will go to the input of the switched side of the relay, and the negative will go to one side of the solenoid.
We also need a connection from +5V, GND and pin 7 of the Arduino to the logic side of the relay, and a connection from the ON switched side of the relay to the other connection on the solenoid.

Now the reason I used standard cables is that I can put all this together and test it (see later) and adjust cables as necessary. Once it’s all been proven to work we can then mount it.

Summary of connections

Arduino Nano pinout requirements
================================
 (See https://forum.arduino.cc/index.php?topic=147582.0 for a reference card)


                      +---------+
                      |         |
                      | o     o | VIN Power +5V in
                      | o     o | GND Ground in
  Ground to relay GND | o     o |
                      | o     o | 5V  5V to relay
                      | o     o |
                      | o     o |
                      | o     o |
                      | o     o |
                      | o     o |
   Signal to Relay D7 | o     o |
                      | o     o |
                      | o     o |
  Serial Pin 3 RX D10 | o     o |
  Serial Pin 2 TX D11 | o     o |
                      | o     o |
                      |         |
                      +---------+

Relay connections
=================
  Control side
     VCC - from  5V on Arduino   
     GND - from GND on Arduino
      IN - from  D7 on Arduino

   Switch side:
     Central pin +5V power in
          ON pin to safe solenoid
                 (if the solenoid opens when the safe should be locked
                  you've picked the wrong side)

Solenoid
========
   (direction doesn't matter)
   +5V from Relay
   GND from power in

RS232
=====
   Pin 2: Connect to Arduino D11
   Pin 3: Connect to Arduino D10
   Pin 5: Connect to power ground  (important!)

Mounting it

This kit is small enough that we can install it inside the cover. I wanted to keep some air gap around the arduino and relay and I realised I could mount them sideways. The easiest way would be to use some folded carboard!

If you’re worried about the cardboard springing loose then you can just put some tape around the ends to add some pressure.

Now I recently got a 3D printer, so I printed a couple of carriers from thingiverse. It does the same thing, but plastic rather than carboard.

Glue the carriers down. It should be below the battery compartment, because this is where the old controller board fitted and so there’s space.

Connect the cables.

You can see I put a cable tie just before the hole; this is an attempt to prevent the cable from being pulled from the outside and breaking a connection.

I decided to add a USB cable (and realised I should have oriented the pieces a different way; there wasn’t enough space for it) and poked the USB cable out through the battery compartment. This allows me to reprogram (upgrade/bug fix) without pulling the whole thing apart. The USB cable fits just inside the battery compartment so it’s not accessible from outside unless the door is open.

Test the setup and then screw the cover back to the door. This hides a multitude of sins and it looks pretty neat!

Arduino software

This sketch should be loaded onto the Arduino. It’s pretty simple and is purely used to control the solenoid. It listens on the serial port at 9600 baud and will respond to a few instructions.

Each instruction begins with a : character. After the instruction is another : and then an optional parameter, and then ends with another : character. No RETURN is needed.

When power is applied the board says “hello” and tells you if the safe is locked or not. Except for the PING command responses will either start with OK or ERROR.

:PING:message:
Response is PINGACK:message:
Used to test communication
:LOCK:password:
Sets the safe to locked state. Can not be used if already locked
:UNLOCK:password:
Unlocks the safe… if the password is correct! Allows for one time use of the OPEN command
:CLEAR:password:
Clears the password and resets the safe to unlocked state.
:TEST:password:
Tests the password but leaves the safe locked.
:OPEN:time:
Opens the solenoid for “time” seconds (default to 5 seconds)
:STATUS::
Returns if the safe is locked or unlocked

Testing

I use Linux and the kermit command, but any terminal emulator would work. The Arduino does not echo what you type, so I’ve turned on local echo mode so I can see what I’m typing. I run the kermit command and then apply power to the safe. You should see the onboard LED start flashing once a second while it is waiting for commands. (You won’t be able to see this when the safe is closed and locked, but it’s useful while testing the setup). Remember, there’s no need to press RETURN (indeed you’ll get an error if you do).

% kermit -l /dev/ttyUSB4 -b 9600
C-Kermit>set local on
C-Kermit>c
Connecting to /dev/ttyUSB4, speed 9600
 Escape character: Ctrl-\ (ASCII 28, FS): enabled
Type the escape character followed by C to get back,
or followed by ? to see other options.
----------------------------------------------------
OK Safe code started.  Safe is unlocked
:ping:1234:PINGACK:1234:
:status::OK Safe is unlocked
:lock:12345:OK Safe locked
:status::OK Safe is locked
:open::ERROR State is locked
:unlock:9876:ERROR Wrong password
:unlock:12345:OK Safe unlocked
:open::OK opening safe for 5 seconds
OK opening safe for 4 seconds
OK opening safe for 3 seconds
OK opening safe for 2 seconds
OK opening safe for 1 seconds
OK completed

When the open command is run then the solenoid should click open and the safe door can be opened. At the end of the time the solenoid should click back again, allowing you to lock the door closed. The open command is also the only command that will return more than one line of results.

Main code

The safe code simply provides control over the solenoid. It effectively just replaces the front panel keypad. But since it’s over a serial port it means that we can write programs for our PC to control it (eg pick a random password). The limits of what the program can do are bounded purely by your imagination. Because the password is stored inside the Arduino, and that’s inside the safe, you can’t cheat… except in one way; you could write a program that will try and brute-force guess the password. Now the safe can handle 100 character long passwords. Let’s assume a standard A-Za-z0-9 character set and 100 guesses per second, that’s 62^100 /100 seconds needed to check every combination. Or 54962310687024123417094664250549034290648513960863640477969506441019257048928403777873869677398592797712143671632918789906836353961913573996848721028517534334752940880822 years.

Good luck with that :-)

I might modify the safe code to sleep for 30 seconds after 3 failed attempts to make it even harder…

My plan is to write a web server interface that will allow for local control of the safe. By making it a web server then normal password managers should also work. Your digital safe’s password can now be stored in the same place as the passwords for your bank account.

Maybe it’s just me, but that idea appeals to me!

I haven’t written this code, yet…

Key man dependencies and resilient processes

Sun, 08 Oct 2017 13:13:50 -0400

Unless you’ve been living in a cave for the past couple of months, you’ll have heard that Equifax, one of the ‘big three’ credit reporting agencies, suffered a massive breach leaking privileged data on over 143 million US people (and millions outside the US as well).

The story went from bad to worse as the company completely failed to handle the response properly, with poor communication, staff giving out the URL to phishing sites, web site failures and the story that three executives sold millions of dollars of shares before the leak notification was made.

As a technologist, I’m more interest on how the breach occurred. Reports from the company are that the web site was susceptible to a Struts2 vulnerability that had been known for months, but hadn’t been patched.

This, naturally, led the question of why it hadn’t been patched. In testimony to Congress, Richard Smith (the Equifax ex-CEO) laid the blame on one unnamed person who failed to require deployment the patch

“The human error was that the individual who’s responsible for communicating in the organization to apply the patch, did not,” Smith said in the hearing.

“So does that mean that that individual knew the software was there,” Rep. Greg Walden replied, “and it needed to be patched, and did not communicate that to the team that did the patching?”

“That is my understanding, sir,” Smith said.

This short segment highlights a number of process failures.

An individual who is responsible
This is an obvious failure mode, and is inherently fragile. It’s almost a textbook example of a “key man dependency.” What if that person was on vacation? Or was sick? Or quit? (Or, for an example of a worst-case scenario, fell under a bus?)
In any critical process (and vulnerability management is a critical process) all steps need to have redundancy in them. This implies a team with cross-checks, a workflow process (new vulnerability reported from upstream, assigned, worked on, tasks initiated… with reporting and alerting if a workflow shows no progress) and so on.
Asset management
To a lot of people, asset management merely means tracking what hardware you have out there. However the true scope is greater than that; it also encompasses software, and it also encompasses components. If this was an inhouse developed application then there needs to be a record of all the opensource components used in the application. This can help from a legal perspective to ensure the license is suitable, but it can also help with identifying applications with vulnerable components. This also implies the need for internal repository services that are used to serve approved code, with firewalls blocking direct access to upstream servers.
Asset discovery
This works in conjunction with the management piece; people make mistakes, developers take shortcuts, vendor products don’t always list their components. Where possible every OS and appliance should be scanned and the discovered components properly catalogued. Workflows can then be spun off to remediate the catalogue (eg license validation). Were you aware that your Oracle product may have Struts2 in it?
Perimeter protection
You do host your applications behind protection technologies such as WAFs, yes? These WAFs must be kept up to date with the latest signatures. Even if you don’t think you host that technology, you may find you are (vendor supplied, developer inclusion) and that your asset reports don’t show it. Vulnerability scanning may not show it, if the path is sufficiently hidden, but that doesn’t mean you don’t have it. The WAF acts as an additional defense layer, and can also be used to help highlight external attack attempts via SIEM reporting.
Internal protection
Related to this would be IDS that may have been able to detect the persistent shell sessions, DAMs to detect anomalous query patterns, multi-tier application architectures (a breach at the presentation layer doesn’t give direct access to the database), sending logs to the SIEM, and so on. This may not have prevented the breach, but may have detected them sooner, before data had been exfiltrated.

Proper deployment of these processes and technologies would have prevented or mitigated the breach; even if the vulnerability hadn’t been patched then the WAF could have blocked access. If neither patching or WAFs worked then IDS and DAM may have alerted and limited the exposure.

This is a classic in-depth scenario, with overlapping solutions in place to mitigate failures elsewhere.

Aside: One thing that would not have helped here, is container technology. I’m getting tired of people telling me “if their app was in a container then they could just redploy with the fixed version”. This wasn’t the problem. Why would someone redeploy containers if they weren’t told?

Summary

We could argue there was a single person to blame for the Equifax issue, but this was not the person Richard Smith suggested. The blame lies at the CTO layer for not ensuring resilient processes and technologies were deployed. Vulnerability Management is a critical function in any organisation that handles PII data; it’s a core requirement of PCI for anyone who handles credit card data.

We’ve also learned how little use auditors are!

Where to run Docker?

Sun, 24 Sep 2017 17:21:45 -0400

I was asked an interesting question:

I am about to investigate Docker. We are moving to AWS too. So in your opinion, should I put energy on EC2 Container services or should I put my energy on Docker on EC2? Which is better ?

I find this type of question interesting because there’s not, really, a “one size fits all” answer. It depends on your use cases.

For example, If you want to use Docker Swarm mode then you’ll need to use standard EC2 VMs and manage the environment in a traditional way. Similarly if you’re dealing with vendor containers and need to use docker-compose or similar.

But you need to be careful with EC2; for example, there’s a rate limiter on the disk I/O and this can hit if you spin up/down containers too quickly.

On the other hand, if you’re creating in-house software from scratch then you might find the Amazon Container Service to be more useful, especially if integrated with Amazon’s secrets management and IAM roles. You definitely get a few useful features…

There’s a downside to using these features, though; you are tying yourself to Amazon. If you use standard docker then you could migrate to any cloud provider that lets you run VMs. Indeed this is one reason why vendors are beginning to distribute containers in Docker format; once you have the Docker engine running then the app should run unchanged.

Another dimension to think about is support; do you want official Docker support? If so, docker EE running on EC2 may be where you go. If you don’t care then Amazon container service may suffice.

Similarly cost; Docker on EC2 means pretty much running an EC2 cluster full time, but Container Service can scale more dynamically… which may save money now that Amazon will charge per-second rather than per-hour.

What about application support processes? Do your developers need access to the host running Docker (something I consider a bad idea, but…). Do your apps need to talk to backend systems that aren’t so forgiving of elasticity? Do your data controls even permit you to run this app in Amazon in the first place?

Summary

I totally expect large organisations will need to run traditional docker installs just because of vendor provided containers and the model they’ll support. You may need Swarm mode; you may need Kubernetes. All of these point to EC2.

But there are some good use cases for Amazon Container Services; it can definitely lead to a more PaaS-like usage of Docker, but with the “tie-in” caveats.

As we can see, really, there’s no “one-size fits all” solution; you may use Container Service for one problem space and Docker EE on EC2 instances for another.

That’s where I think many people will end up.

Docker High Level Challenges with vendor containers

Sun, 17 Sep 2017 14:32:27 -0400

In previous posts I’ve gone into some detail around how Docker works, and some of the ways we can use and configure it. These have been aimed at technologists who want to use Docker, and for security staff who want to control it.

It was pointed out to me that this doesn’t really help leadership teams. They’re getting shouted at; “We need Docker! We need Docker!”. They don’t have the time (and possibly not the skills) to delve into the low levels the way I have. So I’ve put together this higher level overview of the problems with using Docker.

I jokingly call it “Docker Security for CISOs”, and it’s primarily focused on dealing with vendor supplied applications.

Executive Summary

Docker is a virtualization technology that encompasses
- Virtual Hosts
- Virtual Networks
- Virtual Storage
Docker can run any workload
- Web servers, application servers, databases, …
Docker maturity is approximately where Virtual Machine management was 15 years ago

This last point is quite important; we’ve had well over a decade’s worth of experience in dealing with virtual machines and incorporating them into control environments (CMDB, vulnerability management, configuration management, etc). Most of our tooling just doesn’t fit, conceptually, into a container management system (we’re also seeing issues with elastic compute at the VM level). This leads to an important point:

Companies need to prioritize creation of a standard technology stack and support model to enable business lines and dev teams to use this.

Container technology is a form of OS Virtualization

Tools such as VMware ESX virtualize hardware

“Here is a virtual Intel server”
- One physical server, multiple virtual servers
- Each virtual server can be different, but needs to run on the Intel platform
  - Windows and Linux living together
Can run most OS’s unchanged
Can present “virtual” hardware (e.g. network, disk interfaces) that are optimized
- May require special drivers in the guest

Container technologies virtualize the OS

“Here is a virtual Linux instance”
- One Kernel, multiple OS user spaces
- Each OS user space can be different, but needs to run on the Linux kernel
  - Debian and RedHat living together
Can run everything from init/systemd down to a single process

Docker, Docker, Docker. What is Docker?

Docker can be looked at in various different ways…

A container run-time engine
- Uses Linux kernel constructs to segregate workloads
An application packaging format
- Provides a standard way of deploying an application image
A build engine
- Provides a way of creating the application image
An ecosystem
- Repositories, tools, images, workflows, …

Layers upon layers

The Docker engine runs as a privileged process in user space, and so sits on top of a running kernel. This means it can run on a VM or on a bare metal OS deployment. Indeed both forms may exist at the same time. In a VM world the physical host may run both containerized and traditional apps inside different VMs.

It’s important to remember that Docker sits on top of an OS (eg RedHat); it’s not “standalone”

(Image from https://blog.docker.com/2016/04/containers-and-vms-together/ )

Docker container images

A Docker image can contain anything from a single file to a complete OS
- Ubuntu, Debian, CentOS, Alpine, …
This is an opaque box to our existing tooling and processes
- Are there any known vulnerabilities in it?
- What about new vulnerabilities (new CVEs) found after container deployed?
- Our usual agents can not be installed inside the container
A Docker container should be considered immutable
- Don’t patch a running instance, we destroy it and replace it with a fixed copy
- How often does the vendor update their containers with patches?
  - This may result in more pushes to production than a traditional app
A Docker container should be considered stateless
- Stateful information (configuration files, log files, data files…) live outside the container

This impacts how we manage our container environment; so many of our tools and processes just don’t work!

Docker runtime

“Rogue” containers

docker run debian
- We’ve now pulled down a copy of a Debian container from the internet and run it
- Container images could be stored anywhere on the internet
- How do we ensure only approved containers can be run?

Privilege escalation

Giving people access to the docker command grants them root access inside the container
This can also be used to effectively give them root access to the whole machine
Access to the machine may allow people to see config files containing passwords or secrets (API keys, SSL certs, etc) needed by the container application
We may need to consider all interactive access to Docker hosts to be privileged

Image deployment and inventory

How do we ingest new vendor containers?
- What is the process for going from DEV->SIT/UAT->PROD?
- How do we ensure the image promoted to prod is the same as the one in UAT?
- What is the deployment model required for this vendor application
  - Standalone? Compose? Swarm mode? Kubernetes? Something else?
  - What networking constructs are created?
  - What ports are opened?
Inventory of containers
- What images have been approved for use?
  - What have been denied (e.g. too many CRITICAL vulnerabilities)
- Do we know what containers are running, what version, and where?
  - CMDB may be too static, containers may move around a cluster
- Are the containers running that should be running?
  - How do we monitor application availability?
  - Does the deployment include HA designs?

A number of these questions also apply to traditional applications, and can be carried over but I’ve heard senior managers say “it’s a container, just deploy it” bypassing standard code promotion paths.

Some of these questions also apply to virtual appliances (eg applications shipped as an OVA file). Do we have good processes we can adapt, or is this also a gap?

Image integrity and logging

Transient filesystem changes
- A vulnerable app may allow temporary changes to be made inside the container
  - These are lost when the container is restarted
  - Could be used as a pivot point to attack the rest of the network
  - Could be used to host rogue content (phishing site?)
- Existing File Integrity Monitoring tools can not detect this
How does the application log activity?
- Each application may do things differently
- How can we get those logs into Splunk?

Exposed services

One application may effectively introduce a handful of new technologies

A vendor application may consist of multiple containers
- Each container may contain a different open source technology component
  - Tomcat, MySQL, MongoDB, Hadoop, ActiveMQ, Memcached, Zookeeper, …
These services may be exposed onto the main corporate network
- How do we control them?
  - SSL?
  - Usernames/passwords?
  - Encryption of data at rest
  - Backups
How do we monitor access to these services?
- Database Access Monitoring
- Usage logs?

So many options, we can’t support them all

(Source: Docker)

The support model

Docker becomes a critical core infrastructure component
- As critical as the traditional Hypervisor
- There are “community supported” editions and commercial enterprise editions
- The enterprise edition includes additional functionality not in the community edition
Different vendors provide different solutions
- VMware provides “vSphere Integrated Containers” (VIC) and additional tooling
  - Not 100% Docker compatible (e.g. no Swarm Mode)
- Third party support tools exist (management, security, monitoring, etc)
  - Some provide real benefits, others are FUD
The organization needs to determine when it can use open source/community supported tools, and how much vendor enterprise level support is required.
- And then build the tooling and processes around it
- We can use this to also provide a target platform for in-house developed Docker based apps

It’s complicated

Docker was designed to be flexible
Can host anything from a single process to a complete database
- … or even a whole multi-tier solution!
We can’t support every possible deployment model
And let’s not talk about other container ecosystems
- OCI (Open Container Initiative)
- RKT (CoreOS Rocket)
- LXD (Ubuntu)
- OpenVZ
- …

There’s a lot of engineering work that needs to be done!

Summary

This kinda grew a little bit too long for a real CISO deck. It’s maybe 14 pages of powerpoint. But that’s because there’s so much, there!

This isn’t even trying to defend against deliberately bad code (if you can’t trust your vendor, why are you using them?) but against mistakes. Everyone makes them, and this technology is so new.

Just last week I was on a call where the vendor had provided a container with 7 HIGH vulnerabilities in it; we wouldn’t allow a normal app to be deployed that way, so why would we allow this container? It turns out the vendor didn’t have a scanning process in their build pipeline, and so (on my prompting) introduced CLAIR into tooling. Hopefully this will mean less mistakes of this nature and better visibility of new vulnerabilities.

We can’t rely on all vendors doing this, though; we need the tooling inhouse to monitor the environment.

Monitoring my router with graphs

Sun, 03 Sep 2017 17:10:12 -0400

WARNING: technical content ahead! There’s also a tonne of config files, which make this page look longer than it really is, but hopefully they’ll help other people who want to do similar work.

A few months back I replaced my OpenWRT router with a CentOS 7 based one. This machine has been working very well, and handles my Gigabit FIOS traffic without any issues.

I came across a post from Jim Perrin around using tools to collect data and draw graphs on it.

Now I’d started to play with collectd a few years ago, but never went anywhere with it (I still had the source tree from where I compiled it!). Jim’s solution looked simple enough…

He had everything running on the router, but I wanted to keep as much load off the router as possible; let it handle the job of routing packets; databases and graphing can be done elsewhere :-) So I built a small VM (512Mb RAM, 15Gb disk) and called it “monitor”. This also allows me to collect data from other machines and consolidate to one point.

Starting with influxDB

I did things in a slightly different order; I started with the database. That way I could check that collectd was sending data correctly.

So, following Jim’s instructions, I added the repo and installed the software.

% cat /etc/yum.repos.d/influxdb.repo 
[influxdb]
name = InfluxDB Repository - RHEL $releasever
baseurl = https://repos.influxdata.com/centos/$releasever/$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key

% sudo yum install influxdb
[...]

We now need to configure it. I noticed that the defaults log a lot of information; so this configuration turns off all that. You might want to leave it on while building, but beware that your /var/log/messages file will grow large!

The effective configuration is:

reporting-disabled = true
[meta]
  dir = "/var/lib/influxdb/meta"
  logging-enabled=false

[data]
  dir = "/var/lib/influxdb/data"
  wal-dir = "/var/lib/influxdb/wal"
  query-log-enabled=false

[http]
  enabled = true
  bind-address = ":8086"
  log-enabled=false

[[collectd]]
  enabled = true
  bind-address = ":8096"
  database = "collectd"
  typesdb = "/usr/share/collectd"

There are more sections in the file which probably need to be kept there, but everything inside those sections are commented out; they’re the defaults.

Now the collectd section needs to see types.db file from collectd. If you have the collectd RPM installed then this lives in /usr/share/collectd/types.db, otherwise you’ll need to copy the file over. The config file points to the directory the file lives in.

Startup and run the server:

% sudo systemctl enable --now influxdb

% ps -ef | grep influx
influxdb  2597     1  0 Aug31 ?        00:25:34 /usr/bin/influxd -config /etc/influxdb/influxdb.conf

collectd

Now we can configure collectd on the router. Since this is in EPEL, make sure you have the repo enabled and then it’s simple to install:

% sudo yum install collectd

The configuration file is pretty simple:

Hostname    "router"
FQDNLookup   true

BaseDir     "/var/lib/collectd"
PIDFile     "/var/run/collectd.pid"
PluginDir   "/usr/lib64/collectd"
TypesDB     "/usr/share/collectd/types.db"

LoadPlugin syslog
<Plugin syslog>
        LogLevel info
</Plugin>

LoadPlugin cpu
LoadPlugin df
LoadPlugin disk
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin network

<Plugin df>
        ReportInodes true
</Plugin>

<Plugin disk>
        Disk "/^([hsv]|xv)d[a-z][0-9]?$/"
</Plugin>

<Plugin network>
        Server "10.0.0.150" "8096"
</Plugin>

(I’ll come to the “Disk” line a little later).

Let’s start it up!

systemctl enable --now collectd

Verifying it works

At this point collectd should be sending data into InfluxDB. We can check this:

% influx
Connected to http://localhost:8086 version 1.3.5
InfluxDB shell version: 1.3.5
> use collectd;
Using database collectd
> show measurements;
name: measurements
name
----
cpu_value
df_value
disk_io_time
disk_read
disk_value
disk_weighted_io_time
disk_write
interface_rx
interface_tx
load_longterm
load_midterm
load_shortterm
memory_value

Excellent! InfluxDB has a simple SELECT syntax so you can look inside each of the measurements for the raw data.

The default settings for influxDB allows it to retain data for 7 days. If you want to keep data for longer then you’ll need to create a new retention schedule. In my case I wanted 2 months:

> create retention policy two_month on collectd duration 62d replication 1 default;
> drop retention policy autogen on collectd;
> show retention policies;
name      duration  shardGroupDuration replicaN default
----      --------  ------------------ -------- -------
two_month 1488h0m0s 24h0m0s            1        true

Grafana

Let’s head back to Jim’s tutorial… we can now install Grafana on the monitor machine.

% cat /etc/yum.repos.d/grafana.repo 
[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/6/$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

% sudo yum install grafana

Configuration is pretty simple; just set the admin user/password in the [security] section. (We’ll take a look at security in a little bit).

% sudo systemctl enable --now grafana

At this point I was able to go to http://monitor:3000/ and login as the admin user.

We’ll be required to set up a datasource. Because influxDB is on the same machine, we can simply use these settings:

It’s important to make sure the localhost URL is highlighted and not dim.

Creating a dashboard

We’re now ready to create a dashboard to look at all this data. Jim recommended the Host overview dashboard, so that’s where I started.

I realised that some of these graphs were not suitable for a router; for example the networking graphs take the total over all interfaces, which means that “received” and “sent” values are almost identical. After all a router receives data on one interface and sends it back out again on another :-)

The GUI makes it pretty easy to edit panels, move them around, add/remove entries. I wanted the network panels to only refer to br-wan (the interface I use for connecting to the internet). I also didn’t care too much about disk space on this dashboard.

The resulting JSON file is here, if it helps. I can now quickly see my internet usage and CPU load on the server.

Multiple hosts

At this point it becomes very easy to deploy this to other servers. Since my internet hosts are connected via a VPN I can use the same file everywhere. A simple ansible script let me deploy to all the hosts I wanted:

---
- name: Set up collected
  hosts: collectd
  become: False

  tasks:
    - yum: name=collectd state=present

    - service: name=collectd enabled=yes state=started

    - name: collectd.conf
      template: src=Files/collectd/collectd.conf.j2 dest=/etc/collectd.conf owner=root group=root mode=644
      notify:
      - reconfig

  handlers:
    - name: reconfig
      service: name=collectd state=restarted

The “conf” file is identical to the one previously shown, except for the one line:

Hostname    "{{ ansible_hostname }}"

I also added a line for my VM server to collect virt stats:

{% if ansible_hostname == "penfold" %}
LoadPlugin virt
{% endif %}

A quick ansible-playbook command and a few minutes and now I have 7 machines all sending data to influxDB and they now all magically show in the ‘Host Overview” dashboard.

This also explains the odd “Disk” line in the config. The various machines may show hd* or sd* or vd* or even xvd* (Xen client). I also wanted to not collect data for metadisks (md0, md6, etc).

Security

Now this configuration has a lot of things that I’d scream about at work. We’re exposing services to the network without proper access control (primarily influxDB). This means that anyone on my network could send funky data and pollute my database or even read data. There’s no sensitive data stored here, but it’s not a good configuration and would never be acceptable in an enterprise environment.

The tools do allow for authentication and TLS; InfluxDB documentation goes into quite some detail on how to configure this, and both collectd and Grafana can be configured to authenticate to talk to it.

However, I’m not an enterprise; guests are on the guestnet and can’t see this server; I’m not exposing it to the internet…

No doubt this will come back to haunt me at some point :-)

Conclusion

I’m actually not a fan of GUIs for management. This form of “eyes on glass” monitoring means that issues may be lost. However they do have benefits; oddities may stand out.

Indeed, one oddity showed up almost immediately:

Why is my network baseline downloaded 800Mbytes/hour? That seems high!

I was about to hunt down what server was doing a constant 200Kbyte/sec when it struck me… my speed check tester downloads 200Mbytes every 15 minutes; 4 times an hour. That’s 800Mbytes, right there :-)

This graph also clearly shows when I downloaded backups from my internet hosted servers, and when I uploaded my backups to Amazon cloud :-)

Remembering history

Sun, 27 Aug 2017 13:48:37 -0400

“Those who cannot remember the past are condemned to repeat it.” – George Santayana

Default broken

I was reminded, last week, of how old issues repeat.

Back in the 90s it was a truism that if you put an “Out Of The Box” RedHat 4 (not RedHat Enterprise; the original freeware version) server on the internet then it would be compromised within hours. And so we learned; our default builds didn’t have telnet, didn’t have every possible service installed, didn’t have vulnerable configurations. The installer would try and install patches at built time. The root account may not be directly accessed via ssh. Windows machines now have firewalls to block non-local traffic…

Lots of security improvements.

Of course you can put bad machines on the net, but it’s harder to do.

Unfortunately there’s a new generation of developers who failed to learn these lessons. The Mirai botnet spread via Internet Of Things (IoT) devices that failed to do basic security (exposed telnet ports, default unchangeable passwords). This was made worse by using UPnP to expose many of these machines directly onto the internet. These are 20+ year old mistakes, being repeated today.

New ingress points

We learned how to protect environments; firewalls, WAFs, traffic monitoring and so on. We track CVEs, we patch, we have configuration management tools… all in an effort to prevent bad services from being exposed, and to defend against incoming rogue traffic.

However these can only protect against known paths. What happens when a new ingress method is created? For example, IPv6. This can create routes and methods of ingress that bypass existing technologies. For home users used to being behind NAT it may now expose desktops and IoT devices directly to the outside world that were previously protected.

What’s that, you say? No IoT? You don’t have a DVR? A games console? A cellphone? A smart TV? Nothing? Then you’re lucky. So many devices, these days, talk to the internet… and many of those may become directly reachable when IPv6 becomes prevalent.

Cloud

Repeating the mistakes of the past is pretty common when it comes to companies moving into the cloud.

Your traditional corporate datacenter has a lot of technology in place to protect servers within the (virtual) 4 walls of the company. Starting with multi-tier architectures and firewalls (both internal and external), proxies, automation, access controls… companies spend a fortune on technology to help protect servers and data.

And yet it’s not uncommon to see teams deploying servers and software out in the cloud on their own. This “shadow IT” is being created by teams who don’t know (and may not even be aware) of the technology protecting their apps. So they spin up an Amazon account, create networks, expose servers to the internet… and then wonder why they got broken into.

Containers

The same problem applies to container based solutions. We may expose software directly from inside a container (eg a tomcat instance, or something running Apache Struts, or a bad version of OpenSSL), and this is not controlled or even visible to corporate monitoring systems (containers tend to be opaque boxes, and are so dynamic that existing tools may not even be able to handle them, requiring new tools and solutions).

Summary

People deploying solutions need to remember the lessons of the past. We can never go backwards because attacks against those old issues still happen; they’ve been automated into scripts and don’t require any human action to be probed for. Out of curiousity I told my firewall to report incoming port 23 (telnet) connections… I turned the reporting off again a few hours later because the logs were being flooded.

We’re not talking about targeted attacks, this is just random background internet noise. But if you make a mistake then you will be breached and find your server part of a botnet, or a bitcoin miner. If you’re being targeted then these decade old issues will be attempted before the attacker has even sat down with his first cup of coffee.

This isn’t a technology problem, although many companies approach it in that manner. This is an education problem. We need to teach teams why certain processes and procedures exist; why they shouldn’t use shiny-toy-of-the-month; why… Don’t dictate rules, but bring them in as partners.

There will still be tension (“move faster!”) but maybe they’ll be willing to help build the solutions needed to use the shiny-shiny.

Secrets management with Docker Swarm

Sun, 06 Aug 2017 17:34:48 -0400

One of the big problems with a cloudy environment is in how to allow the application to get the username/password needed to reach a backend service (e.g. a MySQL database). With a normal application the application operate team can inject these credentials at install time, but a cloudy app needs to be able to start/stop/restart/scale without human intervention. This can get worse with containers because these may be started a lot more frequently.

It’s tempting to just put the information into a config file that’s part of the container build, but this has a number of issues, including:

The container can be pulled apart and the credentials exposed
The config file may be checked into source control
Passwords may change; you don’t want to rebuild and redeploy each time

The 12 Factor approach is to have credentials passed in via the environment (typically an environment variable). Docker secrets is the model that Docker Swarm Mode uses to achieve this.

Note: this doesn’t work with standalone Docker, nor with containers not started as part of a service. It will work with a single-host swarm and a scale 1 service, though.

What is a secret?

A secret, in Docker, is pretty much any string of text. This means you could put pretty much anything into it; a JSON file, an SSL certificate, API keys, a password… anything. The limit is around 500K, which should be large enough!

This is stored in the Docker Swarm raft log, which means it’s replicated across all the management nodes for resiliency. It’s also (since Docker 1.13) encrypted, which means the secrets are never stored in plain text on any disk.

Creating a secret

This is pretty straight forward. You can specify a file that has the secret, or use - to mean STDIN.

% echo hello | docker secret create my-secret
aqfet0saziaqzspvih6p6w72n

% docker secret ls
ID                          NAME                CREATED             UPDATED
aqfet0saziaqzspvih6p6w72n   my-secret           13 seconds ago      13 seconds ago

% docker secret inspect my-secret
[
    {
        "ID": "aqfet0saziaqzspvih6p6w72n",
        "Version": {
            "Index": 893
        },
        "CreatedAt": "2017-08-06T21:53:38.771368641Z",
        "UpdatedAt": "2017-08-06T21:53:38.771368641Z",
        "Spec": {
            "Name": "my-secret",
            "Labels": {}
        }
    }
]

That’s all there is!

Using secrets

We can create a service that refers to the secret. This will then appear in /run inside the container

% docker service create --detach=true --name secret-service \
         --secret my-secret --entrypoint /bin/sh --tty=true centos
iqdxqenn4hrmo6meobq95rinl

% docker service ps secret-service
ID                  NAME                IMAGE               NODE                   DESIRED STATE       CURRENT STATE                    ERROR               PORTS
kv3uo8qm95h3        secret-service.1    centos:latest       docker-ce.spuddy.org   Running             Running less than a second ago

% docker exec -it secret-service.1.kv3uo8qm95h3anox5q372pojh /bin/sh
sh-4.2# ls /run/secrets
my-secret
sh-4.2# cat /run/secrets/my-secret
hello

Note that /run is a tmpfs filesystem and so the secret still doesn’t appear on disk; it’s only stored in memory.

Using secrets with a stack

Because a stack can have private namespaces as well as using the global one we have to define this as an external resource. The model looks very similar to what we’ve seen for networks. That’s deliberate.

% cat secret_stack.yml
version: '3.1'

services:
  os:
    image: centos
    deploy:
      replicas: 1
    entrypoint: /bin/sh
    stdin_open: true
    tty: true
    secrets:
      - source: my-secret

secrets:
  my-secret:
    external: true

% docker stack deploy -c secret_stack.yml secret-stack
Creating network secret-stack_default
Creating service secret-stack_os

% docker service ps secret-stack
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
eh1ta48g3gfy        secret-stack_os.1   centos:latest       test2.spuddy.org    Running             Running 17 seconds ago

% ssh test2 docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
2e17c5b3a50a        centos:latest       "/bin/sh"           31 seconds ago      Up 30 seconds                           secret-stack_os.1.eh1ta48g3gfyduvgxxx7f0uev

% ssh test2 docker exec secret-stack_os.1.eh1ta48g3gfyduvgxxx7f0uev cat /run/secrets/my-secret
hello

Updating secrets

This is where we start to hit limits. A secret, once in use, can not be changed or even removed. There’s not even a docker secret update option, and docker secret rm fails…

% docker secret

Usage:  docker secret COMMAND

Manage Docker secrets

Options:
      --help   Print usage

Commands:
  create      Create a secret from a file or STDIN as content
  inspect     Display detailed information on one or more secrets
  ls          List secrets
  rm          Remove one or more secrets

Run 'docker secret COMMAND --help' for more information on a command.

% docker secret rm my-secret
Error response from daemon: rpc error: code = 3 desc = secret 'my-secret' is in use by the following services: secret-service, secret-stack_os

This may be a bit of a problem; if you have placed an SSL cert into the secret store and it now needs to be updated then how can you do this?

We have to cheat a little. We can create a new secret and then update the service(s) to use this new secret but under the old name:

% echo A new secret | docker secret create my-secret.v2
kolmhlqxh8wsle75j4siv6dl8

% docker service update --secret-rm my-secret --detach=true \
        --secret-add source=my-secret.v2,target=my-secret secret-service
secret-service

% docker service ps secret-service
ID                  NAME                   IMAGE               NODE                   DESIRED STATE       CURRENT STATE            ERROR               PORTS
4hf3m0rwroeh        secret-service.1       centos:latest       docker-ce.spuddy.org   Ready               Ready 9 seconds ago
kv3uo8qm95h3         \_ secret-service.1   centos:latest       docker-ce.spuddy.org   Shutdown            Running 11 seconds ago

% docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
7ac269ffb300        centos:latest       "/bin/sh"           35 seconds ago      Up 21 seconds                           secret-service.1.4hf3m0rwroehf57evqdowmbsz

% docker exec -it secret-service.1.4hf3m0rwroehf57evqdowmbsz cat /run/secrets/my-secret
A new secret

That’s messy; the docker service update command causes the service to be restarted. In theory that shouldn’t cause downtime to your app, but it’s still a restart and any internal state will be lost.

We also need to update every service that uses a secret before we can remove the old secret

% docker service update --secret-rm my-secret --detach=true \
        --secret-add source=my-secret.v2,target=my-secret secret-stack_os
secret-stack_os

% docker secret rm my-secret
my-secret

% docker secret ls
ID                          NAME                CREATED             UPDATED
kolmhlqxh8wsle75j4siv6dl8   my-secret.v2        6 minutes ago       6 minutes ago

Using secrets as a “seed”

Docker secrets enable us to use an external password vault (e.g. Hashivault). The secret stored inside the Swarm would be sufficient to let your application log into the vault and then get the password it needs (e.g. for the database). It could be used to talk to an API to generate and retrieve an SSL certificate, with the management happening at the enterprise system (e.g. renewal; next time your container starts it will automatically get the new cert).

These “seed” credentials still need to be controlled because they provide access to the final system, but it allows for greater flexibility, especially where credentials may be short lived (Hashivault can generate short-lived passwords in the database; Lets Encrypt certs live 90 days

Since Docker 17.06 there is also docker config. These work conceptually in a similar way to secrets but are implemented slightly differently at the backend; in particular they’re not encrypted at rest and they’re mounted directly into the container.

Management of configs (and the limitations in rotation) are very similar to secrets, just using the docker config command.

Docker configs can work with secrets; e.g. you could create a web server where the config and the HTML index file come from config entries and the SSL certificate comes from the secret entry.

Summary

Docker secrets are currently very limited in what they can do. In particular rotation and updating of secrets is painful and requires your service to be redeployed.

However they do provide a way of handling the “how to get secrets into a container”, which is a hard problem to solve without technology like this.

Using them as a “seed” to access a fully fledged secrets solution (eg a Vault or a Cert management system) overcomes many of the limitations and allows enterprise scale secret management.

Note, however, that anyone in the docker group can get the plain text! This is yet another reason to treat all access to these servers as highly privileged.

Using placement constraints with Docker Swarm

Sun, 30 Jul 2017 13:32:52 -0400

As we’ve previously seen, Docker Swarm mode is a pretty powerful tool for deploying containers across a cluster. It has self-healing capabilities, built in network load balancers, scaling, private VXLAN networks and more.

Docker Swarm will automatically try and place your containers to provide maximum resiliency within the service. So, for example, if you request 3 running copies of a container then it will try and place these on three different machines. Only if resources are unavailable will two containers be placed on the same host.

We saw this with our simple pinger application; it ran on 3 nodes:

% docker service ps pinger
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
xxn5tnej6bao        pinger.1            centos:latest       test1.spuddy.org    Running             Running 5 minutes ago
dcmno194ymjf        pinger.2            centos:latest       test2.spuddy.org    Running             Running 2 second ago
g89lpi6xeuji        pinger.3            centos:latest       test3.spuddy.org    Running             Running 2 second ago

Sometimes, however, you need to control where a container is run. This may be for functionality reasons; for example a container that monitors and reports on the Swarm state needs to run on a manager node in order to get the data it needs. Or there may be OS requirements (a container designed to run on a Windows machine shouldn’t be deployed to a Linux machine!).

Frequently, however, this due to some nodes having the necessary resources, and the most common of these are “volume” dependencies. Remember that while the containers may be run on multiple nodes, a Swarm is really just a collection of standalone Docker engines pulling images from a registry. That means that backend resources, such as file system volumes, are served locally from each node. The contents of /myapp on server test1 may be different from /myapp on test2. We also saw this, in passing, with the MySQL container; it was constrained to only run on a specific node so that the backing datafiles were consistent.

  db:
    image: mysql:5.5
    networks:
      - appdb
    environment:
      - MYSQL_ROOT_PASSWORD=foobar
      - MYSQL_DATABASE=mydb1
    volumes:
      - db-data:/var/lib/mysql
    deploy:
      placement:
        constraints: [node.hostname == test1.spuddy.org]

In this case we use a named volume rather than a filesystem directory, but the constraint is still required; each node would have its own unique “db-data” volume.

Now constraining a host by hostname works, but you’re limited to just a single host. What if you wanted to run 3 copies of Tomcat? We need to constrain it to run, just where the config files are and we can’t just list all three names (the rules are ANDed together).

So, instead, we can define a label and constrain it to that:

  tomcat:
    image: sweh/test:fake_tomcat
    deploy:
      replicas: 2
      placement:
        constraints: [node.labels.Tomcat == true ]
    volumes:
      - "/myapp/apache/certs:/etc/pki/tls/certs/myapp"
      - "/myapp/apache/logs:/etc/httpd/logs"
      - "/myapp/tomcat/webapps:/usr/local/apache-tomcat-8.5.3/webapps"
      - "/myapp/tomcat/logs:/usr/local/apache-tomcat-8.5.3/logs"
    ports:
      - "8443:443"

(“fake_tomcat” is just a dummy program I wrote that listens on the requested port; it doesn’t do any real work).

I want to run this on test1 and test2 so on those two machines I make the necessary directories.

We now need to add the labels to tell the Swarm these are able to run Tomcat:

$ docker node update --label-add Tomcat=true test1.spuddy.org
test1.spuddy.org
$ docker node update --label-add Tomcat=true test2.spuddy.org
test2.spuddy.org
$   docker node inspect --format '{{ .Spec.Labels }}' test1.spuddy.org
map[Tomcat:true]
$   docker node inspect --format '{{ .Spec.Labels }}' test2.spuddy.org
map[Tomcat:true]

I then create the stack, and we can see it running

$ docker stack deploy -c stack.tomcat myapp
Creating network myapp_default
Creating service myapp_tomcat

$ docker stack ls
NAME                SERVICES
myapp               1

$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                   PORTS
bnhvu1cc3orx        myapp_tomcat        replicated          2/2                 sweh/test:fake_tomcat   *:8443->443/tcp

$ docker service ps myapp_tomcat
ID                  NAME                IMAGE                   NODE                DESIRED STATE       CURRENT STATE                ERROR               PORTS
to0yvevap2om        myapp_tomcat.1      sweh/test:fake_tomcat   test2.spuddy.org    Running             Running about a minute ago
dk9pnthgvdcp        myapp_tomcat.2      sweh/test:fake_tomcat   test1.spuddy.org    Running             Running about a minute ago

We can see this is working by testing port 8443:

$ echo hello | nc localhost 8443
You are caller #1 to have reached fake_tomcat on 38be98384cb9:443
fake_tomcat has received message `hello'

$ echo hello | nc localhost 8443
You are caller #1 to have reached fake_tomcat on d095e9f2490f:443
fake_tomcat has received message `hello'

Two calls hit the different containers (as shown by the different hostname in the output).

Multiple services

Let’s create a more complicated stack which has Tomcat and Memcached and Zookeeper.

The additional lines to the stack are:

  zookeeper:
    image: sweh/test:fake_zookeeper
    deploy:
      replicas: 2
      placement:
        constraints: [node.labels.Zookeeper == true ]
    volumes:
      - "/myapp/zookeeper/data:/usr/zookeeper/data"
      - "/myapp/zookeeper/logs:/usr/zookeeper/logs"
      - "/myapp/zookeeper/conf:/usr/zookeeper/conf"
    environment:
      CFG_FILE: /usr/zookeeper/conf/zoo.cfg

  memcached:
    image: sweh/test:fake_memcached
    deploy:
      replicas: 1
      placement:
        constraints: [node.labels.Memcached == true ]

And let’s create the relevant labels to distribute over the three servers. The resulting labels look like:

docker-ce.spuddy.org
    map[]

test1.spuddy.org
    map[Memcached:true Tomcat:true]

test2.spuddy.org
    map[Tomcat:true Zookeeper:true]

test3.spuddy.org
    map[Zookeeper:true]

Note there’s nothing labeled on the manager node; my app won’t run there.

Again, ensure the directories exist with the necessary configuration and deploy:

$ docker stack rm myapp
Removing service myapp_tomcat
Removing network myapp_default

$ docker stack deploy -c stack.full myapp
Creating network myapp_default
Creating service myapp_memcached
Creating service myapp_tomcat
Creating service myapp_zookeeper

$ docker stack ps myapp
ID                  NAME                IMAGE                      NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
iidao6f3vt6z        myapp_zookeeper.1   sweh/test:fake_zookeeper   test2.spuddy.org    Running             Running 34 seconds ago
haq4vvben495        myapp_tomcat.1      sweh/test:fake_tomcat      test1.spuddy.org    Running             Running 35 seconds ago
6u4s6w4fs0cm        myapp_memcached.1   sweh/test:fake_memcached   test1.spuddy.org    Running             Running 37 seconds ago
zixk0eu2aahu        myapp_zookeeper.2   sweh/test:fake_zookeeper   test3.spuddy.org    Running             Running 35 seconds ago
4u8ytws4kl5v        myapp_tomcat.2      sweh/test:fake_tomcat      test2.spuddy.org    Running             Running 35 seconds ago

Rescale

In my fake environment I decided that memcached is running slow (and, besides, only having one copy isn’t very resilient!). Now we can see the power of labels; I can add the Memcached label to another node and then rescale:

$ docker node update --label-add Memcached=true test3.spuddy.org
test3.spuddy.org

$ docker service scale myapp_memcached=2
myapp_memcached scaled to 2

$ docker service ps myapp_memcached
ID                  NAME                IMAGE                      NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
6u4s6w4fs0cm        myapp_memcached.1   sweh/test:fake_memcached   test1.spuddy.org    Running             Running 3 minutes ago
pf56ix45e887        myapp_memcached.2   sweh/test:fake_memcached   test3.spuddy.org    Running             Running 7 seconds ago

Now that was easy because it didn’t have any external volumes to depend on, but we could do the same for zookeeper or tomcat; just create the necessary volumes and configurations on the new node, then add the label.

What if you forget the volumes?

Let’s add Zookeeper to test1 but “forget” to make the volumes:

$ docker node update --label-add Zookeeper=true test1.spuddy.org
test1.spuddy.org
$ docker service scale myapp_zookeeper=3
myapp_zookeeper scaled to 3
$ docker service ps myapp_zookeeper
ID                  NAME                    IMAGE                      NODE                DESIRED STATE       CURRENT STATE                     ERROR                              PORTS
iidao6f3vt6z        myapp_zookeeper.1       sweh/test:fake_zookeeper   test2.spuddy.org    Running             Running 6 minutes ago
zixk0eu2aahu        myapp_zookeeper.2       sweh/test:fake_zookeeper   test3.spuddy.org    Running             Running 6 minutes ago
ytyp3impjzyf        myapp_zookeeper.3       sweh/test:fake_zookeeper   test1.spuddy.org    Ready               Rejected less than a second ago   "invalid mount config for type�
sudgw3w694ja         \_ myapp_zookeeper.3   sweh/test:fake_zookeeper   test1.spuddy.org    Shutdown            Rejected 5 seconds ago            "invalid mount config for type�
3gv64qgli504         \_ myapp_zookeeper.3   sweh/test:fake_zookeeper   test1.spuddy.org    Shutdown            Rejected 6 seconds ago            "invalid mount config for type�

That’s messy!

Eventually the system settles down and runs two copies on an existing node:

$ docker service ps myapp_zookeeper
ID                  NAME                    IMAGE                      NODE                DESIRED STATE       CURRENT STATE                 ERROR                              PORTS
iidao6f3vt6z        myapp_zookeeper.1       sweh/test:fake_zookeeper   test2.spuddy.org    Running             Running 8 minutes ago
zixk0eu2aahu        myapp_zookeeper.2       sweh/test:fake_zookeeper   test3.spuddy.org    Running             Running 8 minutes ago
ib7eiekpes3c        myapp_zookeeper.3       sweh/test:fake_zookeeper   test2.spuddy.org    Running             Running 49 seconds ago

Monitoring of service state becomes very important in this environment!

Migration of services between nodes

For performance reasons I want to move the memcached currently run on test3 and migrate it to test2, so it’s on the same machine as the tomcat instance. We can force a migration of the container by adding the label to the new node and removing it from the old.

$ docker service ps myapp_memcached
ID                  NAME                IMAGE                      NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
6u4s6w4fs0cm        myapp_memcached.1   sweh/test:fake_memcached   test1.spuddy.org    Running             Running 13 minutes ago
pf56ix45e887        myapp_memcached.2   sweh/test:fake_memcached   test3.spuddy.org    Running             Running 9 minutes ago

$ docker node update --label-add Memcached=true test2.spuddy.org
test2.spuddy.org

$ docker node update --label-rm Memcached test3.spuddy.org
test3.spuddy.org

$ docker service ps myapp_memcached
ID                  NAME                    IMAGE                      NODE                DESIRED STATE       CURRENT STATE             ERROR               PORTS
6u4s6w4fs0cm        myapp_memcached.1       sweh/test:fake_memcached   test1.spuddy.org    Running             Running 14 minutes ago
n7bv9331opoq        myapp_memcached.2       sweh/test:fake_memcached   test2.spuddy.org    Running             Running 8 seconds ago
pf56ix45e887         \_ myapp_memcached.2   sweh/test:fake_memcached   test3.spuddy.org    Shutdown            Rejected 14 seconds ago

The instance that had been running on test3 has been rejected and a new instance started on test2. Remember the Docker scheduler will attempt to pick a node that isn’t already running a copy of the container. In this case the only valid servers to meet the placement constraints were test1 and test2, and test1 already had a copy running.

Summary

Using labels allows for a very dynamic way of determining where your workloads run; you can rescale, migrate and even extend the cluster (add new nodes, add labels to the node, modify the scaling) without needing to redeploy the stack.

This can become more important as multiple workloads in multiple stacks are deployed to the same swarm; the stack owners don’t need to know about the underlying node names, they can just use labels and the same stack can be deployed to different targets (how easy would this be to spin up on AWS EC2 instances? No stack changes needed at all!)

Of course this doesn’t come for free; it takes time for containers to spin up (about 6 seconds in my fake_memcached migration) so make sure your services can handle short outages. The closer to 12factor your apps are, the better they are at handling dynamic migration of containers.

And, of course, there’s still the persistent data volume question to handle; this can complicate your deployments.

But despite these complications I would recommend taking a look at labels if you have a need to constrain where your containers run.

A look at Docker Swarm

Sun, 16 Jul 2017 09:24:01 -0400

In my previous entry I took a quick look at some of the Docker orchestration tools. I spent a bit of time poking at docker-compose and mentioned Swarm.

In this entry I’m going to poke a little at Swarm; after all, it now comes as part of the platform and is a key foundation of Docker Enterprise Edition.

Docker Swarm tries to take some of the concepts of a single host model and convert it into a cluster. Some of the same concepts (e.g. networking) are extended to a cluster aware manner (e.g. using VXLAN overlay networks). Where possible it tries to hide this stuff.

Now this entry may look long… but that’s because Swarm does a lot. I’m really only touching on some highlights! There’s a lot more happening behind the scenes.

Complications

Now a Swarm is a collection of Docker Engines. This means that a lot of the stuff we’ve seen in previous entries still applies. In particular, you can still access each node directly and examine the running state. A bit of a gotcha is that you really need a registry to run images.

In previous examples I did something similar to

docker build -t myimage .
docker run myimage

Now this works because the engine has a copy of the image. If I tried to do the docker run command on another machine then it’d fail, because it wouldn’t have a copy of myimage (indeed, it’d try to reach out to docker.io to find a copy there).

There are a few solutions to this; e.g. manually copy images around the cluster, but they don’t really scale. The correct solution is to run a registry, and configure your engines to talk to that.

Docker provides a hosted registry at Docker hub. The free offering is suitable for public storage of images (you can create one private repo, for free). You can pay for additional private repos, which can be configured to require authentication (a number of vendors do this). There is a freebie image that you can pull and run your own internal registry. And there are private registries (eg Docker Datacenter, or VMware harbor); some are commercial. For an enterprise you should look carefully at running your own private repo. For this blog I’m going to use the sweh repo on Docker Hub.

Initializing a Swarm

Once you have your servers built and the Docker software installed and configured (datastores, etc) then we can get started and build a swarm out of it.

On my network I have 4 VMs configured

docker-ce
test1
test2
test3

(my hostnames are so imaginative!). docker-ce will be the manager node, so we can initialize it.

Before we do this, let’s quickly check the state of the network. It should match what we’ve seen before

% docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
20b900380c5e        bridge              bridge              local
9a4091b44793        host                host                local
5e9fc3c20b10        none                null                local

% brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.02420dfebfca       no

% ip -4 addr show dev docker0
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever

Now we can start the swarm.

% docker swarm init
Swarm initialized: current node (f8t7omvmys6pa0nx66wpathuf) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-1234567890123456789069ugkou4sr7k476qlulvi9y3rkpsbt-9ynm7135rlebwycgqnlcap824 10.0.0.164:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

That’s it! We now have a swarm! A swarm of one node, but a swarm.

What does the network look like, now?

% docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
20b900380c5e        bridge              bridge              local
4c524f9b4565        docker_gwbridge     bridge              local
9a4091b44793        host                host                local
7yg0jizx89dt        ingress             overlay             swarm
5e9fc3c20b10        none                null                local

% brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.02420dfebfca       no
docker_gwbridge 8000.024293da454e       no              veth9368d19

% ip -4 addr show dev docker_gwbridge
8: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    inet 172.18.0.1/16 scope global docker_gwbridge
       valid_lft forever preferred_lft forever

So we can see there’s a new bridge network (docker_gwbridge) and an overlay network created.

Now the “swarm token” presented should be kept secret; this is needed to join another Docker node to the swarm. If you don’t keep it secret than anyone (who can reach the manager) can add their machine… and potentially get access to your programs or data.

Fortunately you don’t need to remember this or write it down; you can ask Docker to repeat it

% docker swarm join-token worker
To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-1234567890123456789069ugkou4sr7k476qlulvi9y3rkpsbt-9ynm7135rlebwycgqnlcap824 10.0.0.164:2377

Workers and managers

There are two types of nodes in Swarm

Workers
A worker node is where the compute happens; when you tell Swarm to run 5 copies of your image then it will distribute the jobs across the workers.
Managers
A manager node is also a worker node, so can run jobs. But a manager node is also used to manage the swarm. You can (should!) have multiple managers, for resiliency. Now the managers communicate using the Raft consensus protocol. This means that over half of the managers need to agree on the state of the swarm. Typically that means you need an odd number of managers. With 3 managers you can suffer the loss of 1, and still keep running; with 5 managers you can lose 2; with 7 managers you can lose 3.. picking the right number of managers is an availability question and may depend on the underlying infrastructure “availability zones”.

Side note; managers can be made to not be workers by setting their worker availability to “drain” (docker node update --availability drain), which tells Swarm to remove any running jobs and don’t schedule new ones.

A worker can be promoted to a manager (docker node promote), and a manager can be demoted to a worker (docker node demote).

Adding workers

You simply need to run the docker swarm join command that was mentioned when the swarm was initialized (and that docker swarm join-token worker reports).

% docker swarm join --token SWMTKN-1-1234567890123456789069ugkou4sr7k476qlulvi9y3rkpsbt-9ynm7135rlebwycgqnlcap824 10.0.0.164:2377
This node joined a swarm as a worker.

You can also join a node directly as a manager using the token returned by docker swarm join-token manager

Now my swarm has some compute power!

% docker node ls
ID                            HOSTNAME               STATUS              AVAILABILITY        MANAGER STATUS
k4rec71pftizbzjfcmgio6cz5 *   docker-ce.spuddy.org   Ready               Active              Leader
vbhqixyoe8lcvotwav5dfhf7i     test2.spuddy.org       Ready               Active
wqg2tdlbfow7bftzu5sexz32g     test1.spuddy.org       Ready               Active
zqx3vkx0cdqswlq9q9xn91ype     test3.spuddy.org       Ready               Active

You might have noticed my prompt was %; you don’t need to be root to do any of this if you are in the docker group; membership of this group is very powerful!

If you look at a worker node network it also has the new docker_gwbridge and ingress networks created. But, importantly, the ID of the ingress network is consistent across all nodes. This shows it’s shared across the swarm, whereas the other networks are still local to the node. (Indeed docker_gwbridge is 172.18.0.1/16 on each node).

First service

Docker Swarm works on the concepts of services. These define the unit to be deployed across the swarm. It can be as simple as a single container, or it could be a complicated setup similar to those described by docker-compose.

Let’s just start a copy of centos, and just set it pinging localhost:

% docker service create --detach=true --replicas 1 --name pinger centos ping localhost
vfdxae3ck4r2xt5ig4fd7aqqv

Well, that looks very similar to a docker run command…

We can see the service state, and we can even see what server is running it:

% docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
vfdxae3ck4r2        pinger              replicated          1/1                 centos:latest

% docker service ps pinger
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE                ERROR               PORTS
xxn5tnej6bao        pinger.1            centos:latest       test1.spuddy.org    Running             Running 25 seconds ago

OK, so that’s running on test1. We can login to that machine and use the docker commands we know to look at this:

% ssh test1 docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
3aac8b07c701        centos:latest       "ping localhost"    About a minute ago   Up About a minute                       pinger.1.xxn5tnej6bao96up565ak1xny

% ssh test1 docker logs pinger.1.xxn5tnej6bao96up565ak1xny | tail -3
64 bytes from localhost (127.0.0.1): icmp_seq=160 ttl=64 time=0.017 ms
64 bytes from localhost (127.0.0.1): icmp_seq=161 ttl=64 time=0.347 ms
64 bytes from localhost (127.0.0.1): icmp_seq=162 ttl=64 time=0.048 ms

% ssh test1 docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos              <none>              36540f359ca3        10 days ago          193MB

% ssh test3 docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

%

This should be a big clue that Swarm really just sits on top of the existing docker constructs! test3 doesn’t have the centos image, because it hasn’t needed to pull it, yet.

Re-scale

One container is boring. I want 3 of them!

% docker service scale pinger=3
pinger scaled to 3

% docker service ps pinger
ID                  NAME                IMAGE               NODE                   DESIRED STATE       CURRENT STATE              ERROR               PORTS
xxn5tnej6bao        pinger.1            centos:latest       test1.spuddy.org       Running             Running 4 minutes ago
dcmno194ymjf        pinger.3            centos:latest       test2.spuddy.org       Running             Preparing 14 seconds ago
g89lpi6xeuji        pinger.2            centos:latest       test3.spuddy.org       Running             Preparing 14 seconds ago

% ssh test3 docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos              <none>              36540f359ca3        10 days ago          193MB

We can see state is Preparing, which means it isn’t running yet. Part of this preparation is pulling down the image from the repository. Once the image is down then the container will start up.

But finally:

% docker service ps pinger
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
xxn5tnej6bao        pinger.1            centos:latest       test1.spuddy.org    Running             Running 5 minutes ago
dcmno194ymjf        pinger.2            centos:latest       test2.spuddy.org    Running             Running 2 second ago
g89lpi6xeuji        pinger.3            centos:latest       test3.spuddy.org    Running             Running 2 second ago

The service can be stopped with docker service rm.

Self-healing

Let’s start again, and run 3 new copies of my service:

% docker service ps pinger
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
k9myef61ebqe        pinger.1            centos:latest       test3.spuddy.org    Running             Running 9 seconds ago
quelws25339z        pinger.2            centos:latest       test1.spuddy.org    Running             Running 9 seconds ago
r020adnj1nbd        pinger.3            centos:latest       test2.spuddy.org    Running             Running 9 seconds ago

On test1 I’m going to kill the image:

% ssh test1 docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
e49af88bdfac        centos:latest       "ping localhost"    About a minute ago   Up 59 seconds                           pinger.2.quelws25339z0zkfxek89k5fa
% ssh test1 docker kill e49af88bdfac
e49af88bdfac

We told the Swarm we wanted three replicas, so what did it do?

% docker service ps pinger
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE                ERROR                         PORTS
k9myef61ebqe        pinger.1            centos:latest       test3.spuddy.org    Running             Running about a minute ago
qul4kor2n9j1        pinger.2            centos:latest       test1.spuddy.org    Running             Running 18 seconds ago
quelws25339z         \_ pinger.2        centos:latest       test1.spuddy.org    Shutdown            Failed 24 seconds ago        "task: non-zero exit (137)"
r020adnj1nbd        pinger.3            centos:latest       test2.spuddy.org    Running             Running about a minute ago

Ha! It noticed that one container (pinger.2) had failed and brought up a new one to replace it.

Docker Swarm maintains a history of terminated containers to assist in diagnostics; by default this is something like 5 containers per replica, but can be changed.

Let’s be a little more drastic; hard power down test1.

% docker service ps pinger
ID                  NAME                IMAGE               NODE                   DESIRED STATE       CURRENT STATE                     ERROR                         PORTS
k9myef61ebqe        pinger.1            centos:latest       test3.spuddy.org       Running             Running 7 minutes ago
838i3nnx6ptj        pinger.2            centos:latest       docker-ce.spuddy.org   Running             Starting less than a second ago
qul4kor2n9j1         \_ pinger.2        centos:latest       test1.spuddy.org       Shutdown            Running 6 minutes ago
quelws25339z         \_ pinger.2        centos:latest       test1.spuddy.org       Shutdown            Failed 6 minutes ago              "task: non-zero exit (137)"
r020adnj1nbd        pinger.3            centos:latest       test2.spuddy.org       Running             Running 7 minutes ago

In under a minute the Swarm detected the worker node had failed, and restarted the required job on another node.

This is a great way of doing HA; if you have multiple copies of your service running (scale > 1) then you can handle single server failures almost unnoticed!

The Swarm also tells us the node isn’t available:

% docker node ls
ID                            HOSTNAME               STATUS              AVAILABILITY        MANAGER STATUS
k4rec71pftizbzjfcmgio6cz5 *   docker-ce.spuddy.org   Ready               Active              Leader
vbhqixyoe8lcvotwav5dfhf7i     test2.spuddy.org       Ready               Active
wqg2tdlbfow7bftzu5sexz32g     test1.spuddy.org       Down                Active
zqx3vkx0cdqswlq9q9xn91ype     test3.spuddy.org       Ready               Active

Planned outages

If a server is planned to be taken down (e.g. for patching, rebooting) then the work load can be drained off of it first, and the node made quiescent:

% docker service ps pinger
ID                  NAME                IMAGE               NODE                   DESIRED STATE       CURRENT STATE             ERROR               PORTS
vobdvumj1ve3        pinger.1            centos:latest       test3.spuddy.org       Running             Running 6 seconds ago
q1z3333fmcxa        pinger.2            centos:latest       docker-ce.spuddy.org   Running             Preparing 4 seconds ago
5ddwrc3dcuh8        pinger.3            centos:latest       test1.spuddy.org       Running             Running 1 second ago

% docker node update --availability drain docker-ce.spuddy.org
docker-ce.spuddy.org

% docker node ls
ID                            HOSTNAME               STATUS              AVAILABILITY        MANAGER STATUS
k4rec71pftizbzjfcmgio6cz5 *   docker-ce.spuddy.org   Ready               Drain               Leader
vbhqixyoe8lcvotwav5dfhf7i     test2.spuddy.org       Ready               Active
wqg2tdlbfow7bftzu5sexz32g     test1.spuddy.org       Ready               Active
zqx3vkx0cdqswlq9q9xn91ype     test3.spuddy.org       Ready               Active

% docker service ps pinger
ID                  NAME                IMAGE               NODE                   DESIRED STATE       CURRENT STATE            ERROR               PORTS
vobdvumj1ve3        pinger.1            centos:latest       test3.spuddy.org       Running             Running 47 seconds ago
3ww4npd4yc9g        pinger.2            centos:latest       test2.spuddy.org       Ready               Ready 9 seconds ago
q1z3333fmcxa         \_ pinger.2        centos:latest       docker-ce.spuddy.org   Shutdown            Running 9 seconds ago
5ddwrc3dcuh8        pinger.3            centos:latest       test1.spuddy.org       Running             Running 41 seconds ago

We can see the job that was on docker-ce node has been shutdown and migrated to test2.

Remember, this is a restart of the service, so state internal to the container is lost… but you’re externalizing your state, aren’t you?

After maintenance is complete, the node can be made available again:

% docker node update --availability active docker-ce.spuddy.org
docker-ce.spuddy.org

Load balancer

Docker Swarm has a built in load balancer. Let’s build a simple web server that has a CGI which reports on the hostname. I’m going to push this to docker hub with the name sweh/test. Once it’s pushed I can delete the local version (and the base image I built on)

% docker build -t test .
[...stuff happens...]
Successfully tagged test:latest

% docker tag test sweh/test

% docker push sweh/test
The push refers to a repository [docker.io/sweh/test]
[...stuff happens...]

Let’s run 3 copies of this, exposing port 80. Notice I’m referring to the docker.io container name, so each engine can pull from the repo.

% docker service create --detach=true --replicas 3 --publish 80:80 --name httpd sweh/test
0fsfgeylod67ff4iwx9uiu4i9

% docker service ps httpd
ID                  NAME                IMAGE               NODE                   DESIRED STATE       CURRENT STATE            ERROR               PORTS
v1uwsqfpho2c        httpd.1             sweh/test:latest    docker-ce.spuddy.org   Running             Running 32 seconds ago
40z9wjqhokwq        httpd.2             sweh/test:latest    test1.spuddy.org       Running             Running 9 seconds ago
jwnab0dz2d1s        httpd.3             sweh/test:latest    test2.spuddy.org       Running             Running 19 seconds ago

OK, so I exposed port 80… but what IP address is the service on? The answer is all IP addresses in the swarm. Each node (even if it’s not running the service) will listen on port 80 and direct the request to the a running container.

% curl localhost/cgi-bin/t.cgi
Remote: 10.255.0.2
Hostname: baded58d0501
My Addresses:
    lo 127.0.0.1/8
  eth0 10.255.0.9/16 10.255.0.6/32
  eth1 172.18.0.3/16

% curl localhost/cgi-bin/t.cgi
Remote: 10.255.0.2
Hostname: 69b94822a2f4
My Addresses:
    lo 127.0.0.1/8
  eth0 10.255.0.7/16 10.255.0.6/32
  eth1 172.18.0.3/16

% curl test3/cgi-bin/t.cgi
Remote: 10.255.0.5
Hostname: baded58d0501
My Addresses:
    lo 127.0.0.1/8
  eth0 10.255.0.9/16 10.255.0.6/32
  eth1 172.18.0.3/16

So two calls to localhost hit different containers (different hostnames), and hitting a node not running the service still worked.

This makes it pretty easy to put a HA Proxy (or similar) instance in front of your swarm; it doesn’t matter where your container is running, the combination of HA Proxy and Swarm load balancer means the request will be forwarded.

The underlying constructs used are very similar to those seen previously, on a single node. Some interesting IP addresses in there, which hint at how Docker is doing this magic; note that the primary address on eth0 is unique to the instance, but that a secondary address and the address on eth1 is consistent.

Now we know eth1 is on the docker_gwbridge bridge local to each node. But what is this 10.255.0.0/16 network?

% docker network inspect --format='{{ .IPAM.Config }}' ingress
[{10.255.0.0/16  10.255.0.1 map[]}]

Ah ha! It’s on the ingress network that we’d previously seen flagged consistently across the nodes. Yet this doesn’t appear as a network interface to the host.

This ingress network is the Swarm equivalent of docker0, but using the overlay driver across the swarm. If you look back at the example web server output, you’ll see the “Remote” address was a 10.255.0.x value; we’re seeing the load balancer at work on the ingress network.

Docker Swarm and `docker-compose`

Swarm can not run docker-compose directly. If you try it’ll warn you that you’re just running a single node service. Your compose file will work, but just on the node you started it on; there’s no Swarm interaction.

However, docker stack can read a compose file and create a stack of services out of it.

There are limitations, of course, due to the distributed nature of the run time. A big one, for example, is that filesystem based volumes don’t work so well (/my/dir on the manager may not be present on all the compute nodes!). There are work-arounds (different volume drivers; NFS;…) but this complexity is inherent in a multi-server model.

In earlier examples of docker-compose I created a 3-tier architecture (web,app,DB). Let’s see if we can do the same with Swarm. Now the DB is awkward; it needs access to specific files, so let’s run that on a fixed node.

% cat deploy.yaml
version: "3"

networks:
  webapp:
  appdb:

volumes:
  db-data:

services:
  web:
    image: sweh/test
    networks:
      - webapp
    ports:
      - 80:80

  app:
    image: centos
    networks:
      - webapp
      - appdb
    entrypoint: /bin/sh
    stdin_open: true
    tty: true

  db:
    image: mysql:5.5
    networks:
      - appdb
    environment:
      - MYSQL_ROOT_PASSWORD=foobar
      - MYSQL_DATABASE=mydb1
    volumes:
      - db-data:/var/lib/mysql
    deploy:
      placement:
        constraints: [node.hostname == test1.spuddy.org]

WARNING: This is a really bad setup. db-data is local to each node and so there’s no data persistency if the database is allowed to bounce around the swarm. It’s why we only allow it to run on test1. Do not do this for any production setup (use a better network aware volume setup!); I’m just showing it here for simplicity.

% docker stack deploy -c deploy.yaml 3tier
Creating network 3tier_appdb
Creating network 3tier_webapp
Creating service 3tier_db
Creating service 3tier_web
Creating service 3tier_app

You can see this looks very similar to the previous compose file.

% docker stack ls
NAME                SERVICES
3tier               3

% docker stack services 3tier
ID                  NAME                MODE                REPLICAS            IMAGE               PORTS
7f02afdt674n        3tier_app           replicated          1/1                 centos:latest
qj9ken7d69tq        3tier_web           replicated          1/1                 sweh/test:latest    *:80->80/tcp
u97lwk6kc8cj        3tier_db            replicated          1/1                 mysql:5.5

Of course a stack uses services, so everything we’ve seen still works

% docker service ps 3tier
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
b0kk8dkq7n1i        3tier_app.1         centos:latest       test3.spuddy.org    Running             Running 47 seconds ago
a5wrrwjh07ni        3tier_web.1         sweh/test:latest    test2.spuddy.org    Running             Running 48 seconds ago
fs0owmxbm45h        3tier_db.1          mysql:5.5           test1.spuddy.org    Running             Running 28 seconds ago

It should be no surprise to see the mysql instance running on test1!

Swarm stack networks

We can see the networking type stuff as well. test1 only runs the db layer, so

% ssh test1 docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
1b5cs8cs2sf3        3tier_appdb         overlay             swarm
...

(output edited to remove other networks we’ve already seen)

Similarly, test2 only has the webapp network, but test3 runs the app and so needs both networks:

% ssh test2 docker network ls | grep 3t
s6u8tobcz9rp        3tier_webapp        overlay             swarm

% ssh test3 docker network ls | grep 3t
1b5cs8cs2sf3        3tier_appdb         overlay             swarm
s6u8tobcz9rp        3tier_webapp        overlay             swarm

Again, these networks are overlay networks across the swarm, and they match the bridges we saw for docker-compose.

We can inspect each service to see what IPs have been assigned:

% docker service inspect --format='{{ .Endpoint.VirtualIPs }}' 3tier_web
[{7yg0jizx89dtc9t5kurmpxn95 10.255.0.6/16} {s6u8tobcz9rpcezii6gpuh6c6 10.0.1.2/24}]

% docker service inspect --format='{{ .Endpoint.VirtualIPs }}' 3tier_app
[{1b5cs8cs2sf3zxjo9n40f7g80 10.0.0.4/24} {s6u8tobcz9rpcezii6gpuh6c6 10.0.1.4/24}]

% docker service inspect --format='{{ .Endpoint.VirtualIPs }}' 3tier_db 
[{1b5cs8cs2sf3zxjo9n40f7g80 10.0.0.2/24}]

So the AppDB network is 10.0.0.0/24 and the WebApp network is 10.0.1.0/24. Web also shows on the ingress network because of the port forwarding.

Name resolution works similarly to docker-compse; e.g. from the web instance:

Now this seems to be hitting an issue on my setup; I use 10.0.0.0/24 for my main network (e.g. test1 is 10.0.0.158). There seems to be the occasional slowdown in resolving hostnames (e.g. ping db may take 5 seconds, but then it works).

I’m wondering if the docker instance start to have trouble in routing between the VXLAN and the primary network. So let’s shut this down (docker stack rm 3tier) and modify our stack definition to include network ranges:

networks:
  webapp:
    ipam:
      config:
        - subnet: 10.20.1.0/24
  appdb:
    ipam:
      config:
        - subnet: 10.20.2.0/24

Now when we run the stack we can see the services work as we expect and mirrors what we saw with docker-compose; the web layer can not see the db layer, but the app layer can see both.

  web: eth0=10.20.1.3/24, 10.20.1.2/32   (webapp)
       eth1=172.18.0.3/16                (docker_gwbridge)
       eth2=10.255.0.7/16, 10.255.0.6/32 (ingress)

  app: eth0=10.20.2.3/24, 10.20.2.2/32   (appdb)
       eth1=172.18.0.3/16                (docker_gwbridge)
       eth2=10.20.1.5/24, 10.20.1.4/32   (webapp)

   db: eth0=10.20.2.5/24, 10.20.2.4/32   (appdb)
       eth1=172.18.0.3/16                (docker_gwbridge)

In all cases the default route is to 172.18.0.1 (i.e. the local bridge on each node).

e.g. on the app instance

default via 172.18.0.1 dev eth1
10.20.1.0/24 dev eth2  proto kernel  scope link  src 10.20.1.5
10.20.2.0/24 dev eth0  proto kernel  scope link  src 10.20.2.3
172.18.0.0/16 dev eth1  proto kernel  scope link  src 172.18.0.3

Summary

Argh, this post has grown far too long. I didn’t even talk about secrets management…

Docker Swarm is a powerful extension of the standalone Docker Engine model to a distributed resilient model; run your containers over multiple machines and let Swarm handle the uptime. With stacks it becomes relatively easy to migrate from a one-node compose to a multi-node clustered setup.

Of course there are complications; resources (e.g. directories, repositories) may not be consistent across the cluster. This is inherent complexity in a multi-service solution, and you’ll see similar constraints on other orchestration systems (e.g. Kubernetes). The closer your apps are to 12 Factor the easier it will be to run in a swarm.

I’m not sure why Swarm decided it was free to re-use my primary network range; that definitely caused issues. This probably only a problem with the stack command; if I had manually created networks and attached services this wouldn’t have been a problem. If you’ve seen this issue then please let me know in the comments :-)

Understanding Docker Swarm is important to understanding Docker Datacenter (and Docker Enterprise Edition), since that uses Swarm as the underlying technology.

Simple Docker Orchestration

Sun, 02 Jul 2017 20:27:15 -0400

In earlier posts I looked at what a Docker image looks like and a dig into how it looks at runtime. In this entry I’m going to look at ways of running containers beyond a simple docker run command.

docker-compose

This is an additional program to be installed, but it’s very common in use. Basically, it takes a YAML configuration file. This can describe networks, dependencies, scaling factors, volumes etc etc. Pretty much anything that can be done via the CLI can be described with the compose YAML file. The reason why it is used is that it makes it very easy to describe a reasonably complex run-time and have it all started with a single command.

Building an app and a MySQL database

A typical use case for docker-compose could be to have an application in one container and a database in another. The application could talk to the database and can be scaled independently of the database.

In this example, we’ll just build a generic centos container, rather than an app; this will let us explore stuff.

% cat docker-compose.yaml
version: '2'
services:
  os:
    image: centos
    entrypoint: /bin/sh
    stdin_open: true
    tty: true

  db:
    image: mysql:5.5
    environment:
    - MYSQL_ROOT_PASSWORD=foobar
    - MYSQL_DATABASE=mydb1
    volumes:
    - /tmp/mysqldata:/var/lib/mysql

It’s that simple. As with normal Docker, if the mysql image isn’t present then the daemon will download it automatically.

This is bad practice, hard coding usernames and passwords into files, so don’t do this in real life :-)

What’s useful about this is that the containers are started on a private network so the database can’t be reached from the outside world; only the “app” can see the database.

Let’s start it up! I want two copies of the OS, and one database:

% docker-compose up --scale os=2
Creating network "compose_default" with the default driver
Creating compose_db_1 ...
Creating compose_db_1 ... done
Creating compose_os_1 ...
Creating compose_os_2 ...
Creating compose_os_1 ... done
Creating compose_os_2 ... done
Attaching to compose_db_1, compose_os_2, compose_os_1
db_1  | Initializing database
db_1  | 170602 16:56:52 [Note] Ignoring --secure-file-priv value as server is running with --bootstrap.
db_1  | 170602 16:56:52 [Note] /usr/local/mysql/bin/mysqld (mysqld 5.5.56) starting as process 64 ...
db_1  | 170602 16:56:52 [Note] Ignoring --secure-file-priv value as server is running with --bootstrap.
db_1  | 170602 16:56:52 [Note] /usr/local/mysql/bin/mysqld (mysqld 5.5.56) starting as process 70 ...
[ ... More MySQL startup messages ... ]
db_1  | 170602 16:56:58 [Note] mysqld: ready for connections.
db_1  | Version: '5.5.56'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)

How does this look? What did docker-compose do?

docker-compose created a new bridge (separate from docker0), and connected the three containers to it. Notice that the name of the containers match the entries from the YAML file, with a count. There’s a prefix of “compose”, which is the name of the current directory.

% docker-compose ps
    Name                 Command             State    Ports
-------------------------------------------------------------
compose_db_1   docker-entrypoint.sh mysqld   Up      3306/tcp
compose_os_1   /bin/sh                       Up
compose_os_2   /bin/sh                       Up

% brctl show
bridge name      bridge id               STP enabled     interfaces
br-3f3e56827113  8000.024265371f31       no              veth0e1b53a
                                                         veth55868a8
                                                         vethe42d74c

% docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
3f3e56827113        compose_default     bridge              local

( I edited the output a little, for readability; removed unnecessary data!)

Docker creates resolvable hostnames (internal DNS server) so that each container can reach the other on known names:

% docker exec -it compose_os_1  /bin/sh
sh-4.2# getent hosts os
172.18.0.4      os
172.18.0.3      os

sh-4.2# getent hosts db
172.18.0.2      db

sh-4.2# getent hosts compose_os_2
172.18.0.3      compose_os_2

sh-4.2# ping db
PING db (172.18.0.2) 56(84) bytes of data.
64 bytes from compose_db_1.compose_default (172.18.0.2): icmp_seq=1 ttl=64 time=0.035 ms
64 bytes from compose_db_1.compose_default (172.18.0.2): icmp_seq=2 ttl=64 time=0.047 ms

Since we have two “OS” containers, both of these can be resolved by the “os” hostname (as shown by the getent output). We can also see that each container can be directly named and that there’s an FQDN hiding behind the scenes based on the prefix and the network bridge.

This makes it easy to connect to the database:

With the mysql client inside the OS we can then connect to the database:

sh-4.2# mysql -h db -u root --password=foobar mydb1
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.5.56 MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [mydb1]>

A three tier architecture

We can take this model a step further and create a web/app/db type of solution, purely out of docker-compose:

version: '2'
networks:
  webapp:
    driver: bridge
  appdb:
    driver: bridge

services:
  web:
    image: web-server
    networks:
      - webapp
    volumes:
    - /home/sweh/Docker-Tests/Web-Server/web_base:/var/www/html
    - /home/sweh/Docker-Tests/Web-Server/log:/var/log/httpd
    ports:
    - "80:80"

  app:
    image: centos
    networks:
      - webapp
      - appdb
    entrypoint: /bin/sh
    stdin_open: true
    tty: true

  db:
    image: mysql:5.5
    networks:
      - appdb
    environment:
    - MYSQL_ROOT_PASSWORD=foobar
    - MYSQL_DATABASE=mydb1
    volumes:
    - /tmp/mysqldata:/var/lib/mysql

This defines two bridges (“webapp” is for communication between the web layer and the app layer; “appdb” is between the app and db), and then places each container on the relevant bridges. We only expose port 80 to the main network, which means that only the web service can be accessed from the outside.

We can see the structures this builds (I ran it from a directory called Compose_3_Tier and so the prefix is compose3tier). Firstly the networks. By default they’re a /16 and assigned IP ranges automatically, but this can be overridden in the YAML file.

% docker network ls
NETWORK ID          NAME                  DRIVER              SCOPE
ad325365d45d        compose3tier_appdb    bridge              local
4d3f4c902ff2        compose3tier_webapp   bridge              local

% docker network inspect --format='{{ .IPAM.Config }}' compose3tier_webapp
[{172.18.0.0/16  172.18.0.1 map[]}]

% docker network inspect --format='{{ .IPAM.Config }}' compose3tier_appdb
[{172.19.0.0/16  172.19.0.1 map[]}]

We can see that each container has been given IP addresses on the relevant networks; the “app” container has two addresses, one for each network:

% docker inspect compose3tier_web_1 | grep IPAdd.*172
                    "IPAddress": "172.18.0.3",
% docker inspect compose3tier_app_1 | grep IPAdd.*172
                    "IPAddress": "172.19.0.2",
                    "IPAddress": "172.18.0.2",
% docker inspect compose3tier_db_1 | grep IPAdd.*172
                    "IPAddress": "172.19.0.3",

And we can see that the web layer can see the app layer, but can not see the DB. It can’t even resolve the db hostname!

% docker exec -it compose3tier_web_1 /bin/sh
sh-4.2# ping app
64 bytes from compose3tier_app_1.compose3tier_webapp (172.18.0.2): icmp_seq=1 ttl=64 time=0.069 ms

sh-4.2# ping db
ping: db: Name or service not known
sh-4.2# ping 172.19.0.3
PING 172.19.0.3 (172.19.0.3) 56(84) bytes of data.
^C
--- 172.19.0.3 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 4999ms

Of course this isn’t a true 3-tier network; the DB layer is still NATted to the main network and so can reach the outside world. We’d need to be a little more complicated with out set up. And, of course, it doesn’t have enterprise features (eg packet capture or flow logs between tiers). But it shows us how a single host can use Docker to create a more secure architecture than traditionally done.

Briefly - docker swarm

This was originally an optional extra but (since Docker v1.12) it’s now built in. The idea, here, is to extend the ‘single host’ model of Docker and make it into a cluster. You can add multiple hosts to the “swarm” and Docker will build VXLAN overlay networks between them, much in a similar way that the single host model builds bridges. Containers will be assigned addresses from the overlay network, which means that containers in the same service can talk to each other, even if they’re running on different machines.

Swarm mode can also replace a lot of the functionality of docker-compose, although there are some complications due to the distributed nature of the environment (in the earlier example I used local filesystems as volume mounts… that won’t necessarily work across 10 machines without NFS or a different volume solution!)

Naturally it handles scaling and instance recovery.

Between managers (you can have multiple of them, as well) Docker uses the Raft protocol for consensus.

A feature (since v1.13) is present to handle secrets (e.g. passwords, SSL certs). This data is stored encrypted in the managers. When a container starts up that has been granted access to the secret then the information can be read from /run inside the container (an in-memory filesystem). This is the only place where the secret is unencrypted. Unfortunately these secrets are static in nature; if you change them in the swarm configuration (a complicated process) then the container needs to restart to pick up the changes. This can be done in a rolling nature so it’s not quite as bad as it seems.

I think a good use case for this may be to present an initial credential to allow the container to login to a vault such as HashiCorp vault where the real credential can be stored and managed.

I’m likely to take a deeper look at swarm mode in a later blog entry because it is built into the product and forms the foundation for Docker Enterprise Edition.

Briefly - kubernetes

Kubernetes started as a Google project, based on their internal container scheduler (“borg”). The original name was “Project Seven” (a Star Trek Voyager joke on Seven of Nine; the logo has seven spikes as a nod to this. It need not be limited to just Docker; other runtimes are available. Docker is a very common scenario, though.

Kubernetes takes a different approach to container application management. Here we bundle a group of containers into a pod. This pod is the basic unit of control in Kubernetes, and all the containers inside a pod are co-located on a single host. For resiliency and scalability you deploy more pods across the cluster.

Inside a pod all the containers share a single IP address (unique across the cluster), just listening on different ports. Additional resources (e.g. disk volumes) can also be associated with the pods. As with Docker swarm, these resources need to be available across the cluster.

Multi-tier applications can be defined as a service, which consists of a collection of pods. Kubernetes provides discovery tools via DNS, and can load balance across pods as necessary.

Secrets management is basic, but has gained better access controls in v1.7

Briefly - Mesos

Originally from UCB, this is now an Apache sponsored project. It takes the approach of a “datacenter operating system” (DCOS). Take the concepts of a normal server (management of processes, memory, disk etc etc), and now treat them as a single distributed entity across your datacenter. Of course locality is important for performance, but this can parallel cache coherency on a single machine. You don’t normally care about what CPU your program runs on; should you care what machine it runs on? Let the DCOS handle the hard bits :-)

To this end it provides a number of services, such as “Chronos”; a cron like program to schedule programs across the datacenter.

Originally Mesos used its own container format, but since v0.20 it can now be used to run Docker containers as well.

A large component of Mesos is “Marathon”; this allows Mesos to be used much like a PaaS (Platform as a Service).

I haven’t really spent any real time looking into Mesos; I really only mention it because large companies (Twitter, AirBnB, eBay… even the CIA) use it and it comes up in conversation. I’m not sure I’d want to use this as a general purpose scheduler, but if you have hyper-scale requirements (thousands of servers) then it’s worth taking a look it.

Argl, too many options!

(Source: Docker)

It’s a full time job just keeping up with this! Which is why I’ve only briefly covered some of the solutions, and may have missed something really important and clever (or made mistakes). I focused on docker-compose because it’s simple and powerful, and I’m going to write some more on Swarm mode… but I just don’t have time to run down every alley.

Summary

Docker has grown from a simple container execution engine into a complete ecosystem of products and partner solutions; every day something new comes along. Even the existing stuff changes (eg the new Kubernetes secrets management stuff had changed since I first drafted this out a few weeks back).

Docker also has a lot more complicated functions (security based around SELinux, seccomp) and the ability to have containers share address spaces (for example, you could add a log management container alongside another one, to forward the logs into Splunk).

From an infrastructure perspective it helps to understand how Docker works (which was the driver for this series of blog posts), so that you can understand the consequences of using Docker. I’m a fan of “knowledge in depth” :-)

But, of course, infrastructure has no use if it’s not to run applications. Docker can be used to create more secure solutions (multi-tier in a single host)… but only if we manage it properly. If developers are allowed to put anything they like inside a container then we can end up in a Wild West scenario and expose vulnerable code to external attackers.

Looking at how a Docker container runs

Sat, 24 Jun 2017 09:53:31 -0400

In the previous entry we looked at how a Docker container image is built.

In this entry we’re going to look a little bit about how a container runs.

Let’s take another look at the container we built last time, running apache:

% cat Dockerfile
FROM centos

RUN yum -y update
RUN yum -y install httpd

CMD ["/usr/sbin/httpd","-DFOREGROUND"]

% docker build -t web-server .

% docker run --rm -d -p 80:80 -v $PWD/web_base:/var/www/html \
-v /tmp/weblogs:/var/log/httpd web-server
63250d9d48bb784ac59b39d5c0254337384ee67026f27b144e2717ae0fe3b57b

% docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES
63250d9d48bb        web-server          "/usr/sbin/httpd -..."   2 minutes ago       Up 2 minutes        0.0.0.0:80->80/tcp   modest_shirley

So how does network traffic get into this container? And what does that -p flag mean?

Basic Docker networking

By default, Docker creates a bridge called docker0. This bridge is not connected to the primary network, so there’s no communication to containers on this bridge. The bridge is associated with a private network.

When a container starts up, it is given a virtual ethernet (veth) device, that allows for IP communication between the host and the container. Inside the container it looks just like a normal network device.

This veth device is added to the bridge, and an IP address associated.

With our test Apache container we can see how this looks:

% brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.024234e17ca9       no              veth336564a

% ip -4 addr show dev docker0
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever

%  docker inspect --format='{{ .NetworkSettings.IPAddress }}' modest_shirley
172.17.0.2

So we can see our container’s “veth” device is on the bridge. The bridge itself has an IP address (172.17.0.1) on a /16 network (allowing for 65k addresses). Our container has an address 172.17.0.2 on this network.

We’ve effectively created a private network, 172.17.0.0/16; the host acts as the default gateway for the containers.

Now, of course, the rest of your network (other hosts, etc) do not know how to reach this private network, so a set of iptable rules are created so that outgoing traffic from the container is NAT’d to the host’s IP address. In this way containers can reach out to the main network.

Incoming traffic needs to be port forwarded, and this is set up with the -p flag; you can specify a port on the host and the port on the container it should move to. So -p 80:80 means forward port 80 from the host to port 80 inside the container.

It gets a little messy handling traffic from the outside network to the container, traffic between containers, and traffic from the container to itself

% ps -ef | grep docker-proxy
root     10054   760  0 10:18 ?        00:00:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.17.0.2 -container-port 80

% sudo iptables -v -t nat -L POSTROUTING
Chain POSTROUTING (policy ACCEPT 11 packets, 754 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   78  4961 MASQUERADE  all  --  any    !docker0  172.17.0.0/16        anywhere            
    0     0 MASQUERADE  tcp  --  any    any     172.17.0.2           172.17.0.2           tcp dpt:http

% sudo iptables -v -t nat -L DOCKER     
Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  docker0 any     anywhere             anywhere            
    0     0 DNAT       tcp  --  !docker0 any     anywhere             anywhere             tcp dpt:http to:172.17.0.2:80

Exercise for those following on at home. See what other rules are in the complete iptables output, including the main FORWARD chain

This is just the default; it can be changed!

Container processes

With the CMD entry we told the Docker daemon to start this container by running the httpd process. We know Apache creates a number of child processes. We can see this, pretty easily:

% docker top modest_shirley
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                6458                6442                0                   14:08               ?                   00:00:00            /usr/sbin/httpd -DFOREGROUND
apache              6471                6458                0                   14:08               ?                   00:00:00            /usr/sbin/httpd -DFOREGROUND
apache              6472                6458                0                   14:08               ?                   00:00:00            /usr/sbin/httpd -DFOREGROUND
apache              6473                6458                0                   14:08               ?                   00:00:00            /usr/sbin/httpd -DFOREGROUND
apache              6474                6458                0                   14:08               ?                   00:00:00            /usr/sbin/httpd -DFOREGROUND
apache              6475                6458                0                   14:08               ?                   00:00:00            /usr/sbin/httpd -DFOREGROUND

Note the PIDs are those as seen from the host. When we go inside the container, later, we’ll see the PID numbers look different

Container files

The normal running of program like Apache causes temporary files to be generated (eg the PID file, at the very least). Your app may make use of /tmp, or /run or other areas.

By default Docker running containers are transient; when you shut them down the changes are lost. But while they’re running we can see what changes have been made:

We can see what files have changed

% docker diff modest_shirley
C /run
A /run/mount
A /run/mount/utab
C /run/httpd
A /run/httpd/httpd.pid
A /run/httpd/authdigest_shm.1

Note the log files don’t show because they’re not part of the container image; they were written to a mounted volume (-v flag)

Going inside the container

We’ve seen some ways of looking at a container from the outside, using the docker top and docker diff commands. But what does the container look like from the inside? We can use docker exec to run a command. (The details of how it works involve selecting the same namespaces for your new container, but you can think of it as if you were running a new process inside the container)

The filesystem from inside

% docker exec -it modest_shirley /bin/sh
sh-4.2# ls
anaconda-post.log  dev   lib         media  proc  sbin  tmp
bin                etc   lib64       mnt    root  srv   usr
boot               home  lost+found  opt    run   sys   var

The filesystem looks like a normal CentOS one.

sh-4.2# df -h
Filesystem        Size  Used Avail Use% Mounted on
/dev/mapper/docker-252:1-131082-0940ddeec345786e6a77a45645d662721d239266ede70f2620b21d4abe11ad0d
                   10G  355M  9.7G   4% /
tmpfs             245M     0  245M   0% /dev
tmpfs             245M     0  245M   0% /sys/fs/cgroup
/dev/mapper/dockerce-fs                                                         
                  8.8G   23M  8.3G   1% /etc/hosts
shm                64M     0   64M   0% /dev/shm
/dev/vda3         3.0G  1.6G  1.3G  56% /var/log/httpd
tmpfs             245M     0  245M   0% /sys/firmware

If you look carefully, you can see some “data leakage”. For example, the /var/log/httpd has exposed the filesystem mount point /dev/vda3 (which is where /tmp lives on my test machine). The root disk is showing how much space I allocated to the docker data volume.

Other data may be exposed, eg via the dmesg command

sh-4.2# dmesg | grep Hypervisor
[    0.000000] Hypervisor detected: KVM

We can see that Docker, in its default setup, doesn’t hide so much of the host machine as we might like! That’s the consequence of a virtualised OS, as opposed to virtualised hardware.

Processes from inside

sh-4.2# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 14:08 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache       5     1  0 14:08 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache       6     1  0 14:08 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache       7     1  0 14:08 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache       8     1  0 14:08 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache       9     1  0 14:08 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
root        27     0  1 14:22 ?        00:00:00 /bin/sh
root        31    27  0 14:22 ?        00:00:00 ps -ef

Note the PIDs; the container has its own PID namespace and so our first Apache process now shows as PID 1. Recall, from earlier, that it showed as 6458 in the docker top output.

Networking from inside

This image doesn’t have an ip or ifconfig command inside, but if it did (or if we copied it in) then the output would look something like:

4: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP  link-netnsid 0
    inet 172.17.0.2/16 scope global eth0
       valid_lft forever preferred_lft forever

Similarly the routing table would look like

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         172.17.0.1      0.0.0.0         UG        0 0          0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 eth0

So we can see it shows as a normal network interface, with a default route to the bridge IP address.

Output from a Docker container

A program may send output to stdout or stderr. In a normal VM this might be considered the equivalent of the console. Docker allows us to inspect this as well. Let’s create a simple container that just writes out a line once a minute

#!/bin/sh
while [ 1 ]
do
  echo Hello, the time is `date`
  sleep 1
done

Let’s run this:

% docker run --rm -d timeloop 
c03f1a63e4c7e55ab37c973a2fe231621340c48aae633865049f2588168b1c1e

% docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
c03f1a63e4c7        timeloop            "/hello"            7 seconds ago       Up 2 seconds                            eloquent_torvalds

% docker logs eloquent_torvalds
Hello, the time is Sat Jun 24 14:55:19 UTC 2017
Hello, the time is Sat Jun 24 14:55:20 UTC 2017
Hello, the time is Sat Jun 24 14:55:21 UTC 2017
Hello, the time is Sat Jun 24 14:55:22 UTC 2017

Docker supports different logging modules; we can see what one the container is using:

% docker inspect -f '{{.HostConfig.LogConfig.Type}}' eloquent_torvalds
json-file

This is the default driver and has no default limits; can fill up the disk!

Being nasty

We’ve seen enough in this blog entry to see how we can be nasty. If you look closely, you’ll notice that we did docker exec we were root inside the container. We can abuse this!

sh-4.2# rpm -e passwd
sh-4.2# cat > /bin/passwd
echo Hahahahaha
sh-4.2# chmod 755 /bin/passwd
sh-4.2#

OK, that’s not much of an abuse, but it shows we can make changes.

Fortunately we can detect this type of abuse:

% docker diff modest_shirley
C /root
A /root/.bash_history
[ ... ]
C /etc
C /etc/pam.d
D /etc/pam.d/passwd
C /var
C /var/lib
C /var/lib/rpm
[ ... ]
C /usr/bin
C /usr/bin/passwd
[ ... ]

If we know what files should change (the /tmp and /run files?) then we may be able to use this for intrusion detection (only if filesystem artifacts are left behind) and File Integrity Monitoring (FIM).

Changes are transient

If we destroy and recreate this container then those changes are lost and a “virgin” image is restarted.

% docker kill modest_shirley
modest_shirley

% docker run --rm -d -p 80:80 -v $PWD/web_base:/var/www/html -v /tmp/weblogs:/var/log/httpd web-server
2118033d42f2fe6bfe10861e838adb7a5df0c408431ba070d77ff6fa213ff45d

% docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES
2118033d42f2        web-server          "/usr/sbin/httpd -..."   3 seconds ago       Up 3 seconds        0.0.0.0:80->80/tcp   compassionate_liskov

% docker diff compassionate_liskov
C /run
C /run/httpd
A /run/httpd/authdigest_shm.1
A /run/httpd/httpd.pid

This is useful for recovering from a broken container, but it loses a potentially useful audit trail (which could hamper incident response).

Keeping changes after termination

We can keep terminated containers by not using the --rm flag, but this will start using up disk space.

To demonstrate this I created a simple container that just creates three files and terminates (by now you should be able to do this yourself, so I won’t show the Dockerfile or script).

We’ll run it without the --rm flag:

% docker image ls change
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
change              latest              7627c3a09d4e        30 minutes ago      124 MB

% docker run change

% docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

The container doesn’t show in the ps listing. We need to use an additional flag to show these terminated containers:

% docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                     PORTS               NAMES
b216fbcca2c0        change              "/hello"            7 seconds ago       Exited (0) 6 seconds ago                       naughty_sinoussi

There it is!

Because the results have been kept around we can inspect it and even pull the contents.

% docker diff naughty_sinoussi
C /tmp
A /tmp/testfile2
C /run
A /run/testfile3
A /testfile1

% docker cp naughty_sinoussi:/tmp/testfile2 - | tar tvf -
-rw-r--r-- 0/0              29 2017-06-09 14:40 testfile2

% docker cp naughty_sinoussi:/tmp/testfile2 - | tar xOf - testfile2
I am modifying a file in tmp

The docker cp command is useful; it can be used to extract (or push!) files and directories from a container. The output is in a tar format. You can do this on running containers as well.

Finally we can clear this up:

% docker rm naughty_sinoussi
naughty_sinoussi

Disk space used

Obviously keeping these changes (and “console” log output) around takes up disk space. But how much?

Let’s start with a clean system:

% docker info | grep Space.Used
 Data Space Used: 840.2 MB
 Metadata Space Used: 1.217 MB

% docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              10                  0                   660.8 MB            660.8 MB (100%)
Containers          0                   0                   0 B                 0 B
Local Volumes       0                   0                   0 B                 0 B

Now let’s run the docker run change command 100 times (obviously by cheating and run it in a loop).

Now how much space is used?

 Data Space Used: 1.45 GB
 Metadata Space Used: 6.554 MB

Images              10                  1                   660.8 MB            660.8 MB (99%)
Containers          100                 0                   8.8 kB              8.8 kB (100%)
Local Volumes       0                   0                   0 B                 0 B

Now I know my script changes around 88 bytes of data inside the container. The df command only shows a 8.8K increase in size (which matches 88 bytes changed in 100 containers), but the info command shows usage has grown by over 600M

Important note: The df command is slow with so many terminated containers

This gives us the ability to create a management process around running containers. For example we could start them up without the --rm flag. Periodically during the running we can check the docker diff results and if something looks bad we can alert the SOC. Potentially terminate the container. Similarly on container termination we can check the diff results and if that looks clean then we can rm the results to recover disk space, or else retain it for forensic analysis.

Read-only containers

There’s another way of running Docker that can help protect against modification: use the --read-only flag. With this the whole filesystem is made immutable. Now your normal app requires some temporary space; we can do this with --tmpfs. Annoyingly the permissions on /run may not be correct, so we can create a simple startup wrapper.

Going back to our Apache example, we build it the same way but with a startup wrapper instead

% cat startup
#!/bin/sh
mkdir -m 0777 /run/httpd
exec /usr/sbin/httpd -DFOREGROUND

% cat Dockerfile
FROM centos
RUN  yum -y update
RUN  yum -y install httpd
ADD /startup /
CMD ["/startup"]

% docker build -t readonly-web .

% docker run -d --rm --read-only -p80:80 -v $PWD/web_base:/var/www/html:ro -v /tmp/weblogs:/var/log/httpd --tmpfs /run --tmpfs /tmp readonly-web

Note the :ro on the /var/www/html directory to make the html tree also immutable, and /run and /tmp are set as tmpfs directories

If we try to make changes inside the container it fails, but the web server can still write out its logs

% docker exec -it f17484a0529f /bin/sh
sh-4.2# touch /foo
touch: cannot touch '/foo': Read-only file system

sh-4.2# rpm -e passwd
error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Read-only file system)

sh-4.2# touch /tmp/foo

sh-4.2# touch /var/www/html/bar
touch: cannot touch '/var/www/html/bar': Read-only file system

sh-4.2# tail -1 /var/log/httpd/error_log
[Mon Jun 12 17:01:11.024721 2017] [core:notice] [pid 1] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

A bit of a gotcha, here, is that docker diff shows no files have changed, which may hide intrusion indicators! We’ve been inside the container and looked around, but no history file was generated. This may be a small price for preventing abuse in the first place.

Being nasty outside the container

We can take what we’ve learned and use that to break into the host.

For example, we could map the root directory!

% docker run --rm -it -v /:/mnt centos /bin/sh
sh-4.2# cp /bin/id /mnt/tmp/badperson
sh-4.2# chmod 4711 /mnt/tmp/badperson
sh-4.2# exit
exit

% id
uid=500(sweh) gid=500(sweh) groups=500(sweh),499(docker)

% /tmp/badperson
uid=500(sweh) gid=500(sweh) euid=0(root) groups=500(sweh),499(docker)

Note the euid has changed.

If you give a normal user permission to run the docker command (which, basically, means being in the docker group) then they have effective root on the whole machine.

SELinux can mitigate this, to some extent, by preventing the container from having permissions to modify stuff. Indeed, I had to disable SELinux to do this test. These security features are there for a reason, but sometimes they’re disabled.

The best defense is to not allow people to be in the docker group in the first place!

Summary

In this blog entry we’ve looked at the running container:

Networking, Processes, file changes
Container stdout/stderr logs
Abusing docker privileges (root exploit!)
- And how we can detect this
- Some protections we can do against this

Docker also has a lot of advanced security functions (SELinux, AppArmour, seccomp, capabilities) which can protect the system and the application. These are beyond the scope of this “basics” blog entry, but are definitely something an enterprise user of Docker needs to be aware of.

What is a Docker container?

Sun, 18 Jun 2017 16:23:36 -0400

Container technology, specifically Docker, is becoming an important part of any enterprise. Even if you don’t have development teams targeting Docker you may have a vendor wanting to deliver their software in container form. I’m not so happy with that, but we’re going to have to live with it.

In order to properly control the risk around this I feel it helps to have a feeling for the basics of what a Docker container is, and since I come from a technical background I like to look at it from a technology driven perspective.

This is the first in a series of “Docker basics” that will try to explain the what and the why’s of a Docker container.

Container technology is a form of OS virtualisation

We’re used to virtualisation in the form of VMware or Xen or KVM (or LDOMs on Solaris, or LPARs on AIX).

Tools such as VMware ESX virtualize hardware

“Here is a virtual Intel server”
- One physical server, multiple virtual servers
- Each virtual server can be different, but needs to run on the Intel platform
  - Windows and Linux living together
Can run most OS’s unchanged
Can present “virtual” hardware (e.g. network, disk interfaces) that are optimized
- May require special drivers in the guest

On the other hand, container technologies virtualize the OS

“Here is a virtual Linux instance”
- One Kernel, multiple OS userspaces
- Each OS userspace can be different, but needs to run on the Linux kernel
  - Debian and RedHat living together
Can run everything from init/systemd down to a single process

Docker, Docker, Docker. What is Docker?

Docker can be thought of in a number of different ways

A container run-time engine
- Uses Linux kernel constructs
  - LXC
  - Namespaces
  - cgroups
  - Networking
An application packaging format
- Provides a standard way of deploying an application image
  - The “container” contains the code, dependencies, etc inside the image
A build engine
- Provides a way of creating the application image
An ecosystem
- Repositories, tools, images, workflows…

Layers upon layers

(Image from https://blog.docker.com/2016/04/containers-and-vms-together/ )

The worlds simplest container

Let’s get down to basics and build our first container. First a simple “Hello world” program in C:

% cat hello.c
#include <stdio.h>
#include <stdlib.h>

int main(int argc,char *argv[])
{
  printf("Hello, World\n");
}

We compile it static (which we’ll get to in a moment)

% gcc -static -o hello hello.c

To build a Docker image we can use a Dockerfile, which contains the instructions:

% cat Dockerfile
FROM scratch
COPY hello /
CMD ["/hello"]

This file means

“start from a blank (scratch) image”
copy the file “hello” into the root directory of the container.
When the container is started run the command “/hello”.

We can now tell the system to build the container:

% docker build -t hello .
Sending build context to Docker daemon 852.5 kB
Step 1/3 : FROM scratch
 --->
Step 2/3 : COPY hello /
 ---> ff932a2c2088
Removing intermediate container 60fa0962d256
Step 3/3 : CMD /hello
 ---> Running in a6375caaec44
 ---> 1cb4c0f3e212
Removing intermediate container a6375caaec44
Successfully built 1cb4c0f3e212

An important note, here, is that the engine does the build (“Sending build context to Docker daemon”). Basically the command will tar up the current directory, send that to the engine, the engine does all the hard work and creates the image. It is not done in your current shell.

We can see the results, and run the container:

% docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
hello               latest              1cb4c0f3e212        4 seconds ago       849 kB

% docker run --rm hello
Hello, World

The --rm flag means “clean up after running”. We’ll see more about this in later blog entries.

What does the container image look like?

So now we’ve built this container, what does it look like?

We can pull the container from the engine and look at it

% mkdir /tmp/container
% cd /tmp/container
% docker save hello | tar xvf -
1cb4c0f3e212f5abfb4b0b74d36a6ace8b7fad164b3e460d8cbf7fb1c3905270.json
59aef640ce511eadb6169072a6d8cefd95a3d1b14b73b92bd86fcb7e0b67618e/
59aef640ce511eadb6169072a6d8cefd95a3d1b14b73b92bd86fcb7e0b67618e/VERSION
59aef640ce511eadb6169072a6d8cefd95a3d1b14b73b92bd86fcb7e0b67618e/json
59aef640ce511eadb6169072a6d8cefd95a3d1b14b73b92bd86fcb7e0b67618e/layer.tar
manifest.json
tar: manifest.json: implausibly old time stamp 1970-01-01 00:00:00
repositories
tar: repositories: implausibly old time stamp 1970-01-01 00:00:00

The content is in the layer tar file

% tar tf 59aef640ce511eadb6169072a6d8cefd95a3d1b14b73b92bd86fcb7e0b67618e/layer.tar
hello

It’s basically just a tar ball with meta data!

Why did I build that with -static?

In the above example I used -static while compiling. Let’s see what happens if I don’t do that.

% gcc -o hello hello.c

% docker build -t hello-dynamic .
Sending build context to Docker daemon 12.29 kB
Step 1/3 : FROM scratch
 --->
Step 2/3 : COPY hello /
 ---> b94d1686eec2
Removing intermediate container f5fdec11138e
Step 3/3 : CMD /hello
 ---> Running in bc887d4cc424
 ---> 5f13658614ea
Removing intermediate container bc887d4cc424
Successfully built 5f13658614ea

% docker run --rm hello-dynamic
standard_init_linux.go:178: exec user process caused "no such file or directory"

“No such file or directory” isn’t a helpful error message, but experience with older technologies (chroot environments) reminds me that we need to include all the libraries we need inside the image as well. Remember, a container needs to include all the dependencies it needs!

OK, we can work out what is needed, and update our build and Dockerfile to include what is necessary:

% ldd hello
        linux-vdso.so.1 =>  (0x00007ffca81c9000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fa02da4d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa02de22000)

% mkdir lib64

% cp /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2 lib64

% cat Dockerfile
FROM scratch
COPY hello /
COPY lib64/ /lib64/
CMD ["/hello"]

If you ever built an “anonymous FTP” chroot then this would feel familiar; all library dependencies need to be inside the container image.

This time the container runs:

% docker build -t hello-dynamic .
Sending build context to Docker daemon 2.288 MB
Step 1/4 : FROM scratch
 --->
Step 2/4 : COPY hello /
 ---> Using cache
 ---> b94d1686eec2
Step 3/4 : COPY lib64/ /lib64/
 ---> 280171c871b9
Removing intermediate container 3872e77dd215
Step 4/4 : CMD /hello
 ---> Running in fe399496bd09
 ---> 6d706ccc02fc
Removing intermediate container fe399496bd09
Successfully built 6d706ccc02fc

% docker run --rm hello-dynamic
Hello, World

What does this new container look like?

We’d expect to see the 3 files inside the layer file. Let’s look:

% mkdir /tmp/container
% cd /tmp/container
% docker save hello-dynamic | tar xvf -
10e6ef63539659eef459b5b0795ad1d91e44b4d3d6b5cf369f026cabbef333f4/
10e6ef63539659eef459b5b0795ad1d91e44b4d3d6b5cf369f026cabbef333f4/VERSION
10e6ef63539659eef459b5b0795ad1d91e44b4d3d6b5cf369f026cabbef333f4/json
10e6ef63539659eef459b5b0795ad1d91e44b4d3d6b5cf369f026cabbef333f4/layer.tar
1f5263a466b60441b1569ae0bf6d059f6a866d721bde8c8e795313116400fb4f/
1f5263a466b60441b1569ae0bf6d059f6a866d721bde8c8e795313116400fb4f/VERSION
1f5263a466b60441b1569ae0bf6d059f6a866d721bde8c8e795313116400fb4f/json
1f5263a466b60441b1569ae0bf6d059f6a866d721bde8c8e795313116400fb4f/layer.tar
6d706ccc02fc6c6ead9b6913b8bd21f15a34abcb09b5deb8e5407af5f4203f14.json
manifest.json
tar: manifest.json: implausibly old time stamp 1970-01-01 00:00:00
repositories
tar: repositories: implausibly old time stamp 1970-01-01 00:00:00

Oh, hey, multiple layers! Let’s look inside each layer:

% tar tf 10e6ef63539659eef459b5b0795ad1d91e44b4d3d6b5cf369f026cabbef333f4/layer.tar
hello

% tar tf 1f5263a466b60441b1569ae0bf6d059f6a866d721bde8c8e795313116400fb4f/layer.tar
lib64/
lib64/ld-linux-x86-64.so.2
lib64/libc.so.6

So each COPY command created a new layer to the Docker image, which show up in the manifest file

% jq '.[] | [ .Layers ]' < manifest.json
[
  [
    "10e6ef63539659eef459b5b0795ad1d91e44b4d3d6b5cf369f026cabbef333f4/layer.tar",
    "1f5263a466b60441b1569ae0bf6d059f6a866d721bde8c8e795313116400fb4f/layer.tar"
  ]
]

This is hard work! What if I wanted perl or php or…?

This working out of dependencies isn’t a good solution. Each library may have its own set of dependencies, and libraries that are optionally loaded at run time (e.g. name resolver libraries) won’t get found. So, instead of starting from scratch we can bring in a base OS image instead:

% cat Dockerfile
FROM debian:latest
COPY hello /
CMD ["/hello"]

The FROM debian:latest line tells the engine to use an image of that name as the base layer. If the engine doesn’t know about this then it will reach out to a repository to find and pull the image (default: docker.io). This is done by the engine and the engine can be configured to use HTTP proxies (if necessary). So if you don’t explicitly block stuff then any developer can pull almost anything in.

This layer now becomes the base on which our code gets copied. The build now runs pretty much as expected; we can see the engine doing the pull to get the necessary image.

% docker build -t hello-debian .
Sending build context to Docker daemon 12.29 kB
Step 1/3 : FROM debian:latest
latest: Pulling from library/debian
10a267c67f42: Pull complete
Digest: sha256:476959f29a17423a24a17716e058352ff6fbf13d8389e4a561c8ccc758245937
Status: Downloaded newer image for debian:latest
 ---> 3e83c23dba6a
Step 2/3 : COPY hello /
 ---> c76829938202
Removing intermediate container 78158e94a68c
Step 3/3 : CMD /hello
 ---> Running in 83c90a1cb1d8
 ---> 740546dadcc9
Removing intermediate container 83c90a1cb1d8
Successfully built 740546dadcc9

% docker run --rm hello-debian
Hello, World

Size and complexity

This convenience doesn’t come for free, of course!

% docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
hello-debian        latest              740546dadcc9        3 minutes ago       124 MB
hello-dynamic       latest              6d706ccc02fc        17 hours ago        2.28 MB
hello               latest              1cb4c0f3e212        17 hours ago        849 kB
debian              latest              3e83c23dba6a        3 weeks ago         124 MB

Our “Hello, World” application has grown from 849 Kbytes to a whopping 124Mbytes.

% docker save hello-debian | tar xf -
tar: manifest.json: implausibly old time stamp 1970-01-01 00:00:00
tar: repositories: implausibly old time stamp 1970-01-01 00:00:00

% for a in */layer.tar; do print -n "$a: " ; tar tf $a | wc -l ; done
7a83b1430fe3eaed26ecd9b011dcd885c6f04e45bee3205734fea8ce7382a01e/layer.tar: 1
a1ae6cd0978a7d90aea036f11679f31e1a3142b1f40030fb80d4c8748b0a6a01/layer.tar: 8259

It’s also grown from 1 file to 8,260 files.

But we no longer need to track dependencies. Is this a win? Well, from a developer’s pespective, it most definitely is.

Security concerns

From a security perspective though… your application image now has a complete(ish) copy of Debian inside it. This is hidden from your traditional infrastructure operations team. It may not even use a standard OS used by your organisation! If there’s a patch in the OS layer then the application team are now responsible for patching and deploying the fixed containers.

Will they be tracking these vulnerabilities? How quickly would they have fixed shellshock? Or (in more advanced cases) Apache Struts?

Advanced builds

If we’re pulling in upstream images we can use their packages as well

% cat Dockerfile
FROM centos

RUN yum -y update
RUN yum -y install httpd

CMD ["/usr/sbin/httpd","-DFOREGROUND"]

% docker build -t web-server .
...[stuff happens]...

The RUN command can run any shell script, and creates a new layer. In this case I created two layers (one for yum update and one for yum install). I could have done a more complicated script such as yum update && yum install and only created one layer.

Exercise for those following at home: Extract the image (docker save) and look at the layers created. Note that data in /var/lib/yum and /var/lib/rpm has also changed.

But we now have a web server! Let’s give it a file to serve:

% cat web_base/index.html
<html>
<head>Test</head>
<body>
This is a test
</body>
</html>

% docker run --rm -d -p 80:80 -v $PWD/web_base:/var/www/html \
-v /tmp/weblogs:/var/log/httpd web-server
63250d9d48bb784ac59b39d5c0254337384ee67026f27b144e2717ae0fe3b57b

The -v flag maps the local directory to a volume inside the container. In this case we mapped the HTML directory and the apache log directory

This is one way we can pass configuration to a container and get log information out of a container

Notice we didn’t write any special code! This is standard apache running in a container

And it works!

% docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES
63250d9d48bb        web-server          "/usr/sbin/httpd -..."   2 minutes ago       Up 2 minutes        0.0.0.0:80->80/tcp   modest_shirley

% curl http://localhost
<html>
<head>Test</head>
<body>
This is a test
</body>
</html>

% cat /tmp/weblogs/access_log
172.17.0.1 - - [18/Jun/2017:14:08:59 +0000] "GET / HTTP/1.1" 200 63 "-" "curl/7.29.0"

In the ps command we can see the port mapping.

The name is randomly generated, but can be set by the docker run comamnd line

Summary

In this blog we’ve taken a basic look at:

The Docker engine
How to build a simple container
- What the container looks like
Library dependencies
- Layers
Pulling in external images
Running almost anything at build time (“yum”)
Mapping local directories into the container
- Configuration files, log files

We’ve also seen some security concerns around developers being able to pull random files from anywhere into these containers, and it being an opaque box to infrastructure operate.

We can start to see some of the risks in running Docker instances. These also apply to vendor provided containers, and for similar reasons. The vendor becomes responsible for patching the contents of the container, which should mean more frequent releases (they not only need to fix their application, they also need to fix the OS). Do they? Will your teams have the time needed to run these new containers through test/qa/prod release cycles?

In later blog entries I’ll start to look inside a running container and look at some ways we can detect intrusion and protect against it.

Introduction to web SSL certificates

Sun, 11 Jun 2017 16:29:07 -0400

Last year I wrote about how I used Letsencrypt to handle the SSL certificates for this site. In this entry I’m going to take a step back and discuss the basics of what an SSL certificate is and the steps involved in managing them. There’s a lot of jargon involved, which can make this seem more complicated than it already is.

Note that in this post I’m likely to use the words “SSL” and “TLS” interchangeably. TLS is the successor to SSL (in some respects you can think of TLS1 as SSL4) and many of the same things apply. The basics of certificates are the same. I started playing with SSL over 20 years ago, and tend to use that term (even when I should really use TLS).

Note also you may see the phrase “X509 certificate” used in some places. The format typically used to manage SSL certificates is X509; you may say that an SSL cert is an X509 cert with the “server authentication” attribute. X509 certs may also be used elsewhere (e.g. for code signing), but if you’re reading some SSL tutorial and it tells you to do stuff with the X509 file then we’re talking about the same thing.

SSL certs can be present in various services (e.g. SMTPS, IMAPS, NNTPS) but the most common use case most people see are for HTTPS (encrypted web sites, like this one). I’ll describe things in those terms, but the same concepts apply.

What is an SSL certificate used for

An SSL cert does two things for you:

Identify the server
Provides a means of performing public key encryption with the server

So what this means is that when you go to this site your browser will be told “Hi, I’m www.sweharris.org!”, and will be given a way of verifying that the certificate is not lying to you.

It’s important to note that the certificate doesn’t tell you who runs the site but we’ll see, later, how this can be handled.

How is it generated

Modern tools such as Letsencrypt can hide all this from you, but it may be useful to know the underlying basics.

We start off by generating a public/private key pair. Typically, today, this would be an RSA 2048 or 4096 bit key. As with all public key systems the private key is the important part and should be kept protected.

We then create a Certificate Signing Request (CSR). This contains the public part of the key pair, along with some information about the site.

The common name (CN) of the site (stored in the Subject field)
Other names the website may use (Subject Alternate Names - SAN)
(Optionally) other information such as the city/state/country or the name of the site owner (also stored in the Subject field)
A signature for the request (typically SHA256 based).

So, for example, this web site has

    Public Key: (4096 bit RSA key)
       Subject: CN=www.sweharris.org
           SAN: DNS:www.sweharris.org, DNS: sweharris.org
     Signature: (SHA256 sig)

The certificate for www.google.com has:

    Public Key: (2048 bit RSA key)
       Subject: C=US, ST=California, L=Mountain View, O=Google Inc, CN=www.google.com  
           SAN: DNS:www.google.com
     Signature: (SHA256 sig)

Other fields can be added to this request, but these are the minimum.

This CSR is then sent to the Certificate Authority (CA) for signing.

NOTE: As you can see, the CN field isn’t too well defined, whereas the SAN field is better organised. Recent versions of Chrome have started to require SAN entries and will only use those to validate the site.

What is certificate signing

Once the CA receives the CSR it needs to validate the request is correct (am I allowed to request a certificate for www.sweharris.org, or is this a naughty person trying to pretend to be me).

Once the CA is happy the request it takes the CSR adds various fields (most importantly a serial number, a start/end date, its own Issuer field, and for what purpose the certificate can be used for). The result is then cryptographically signed (basically a hash is taken of the data, and encrypted with the CA’s own private key). The result is returned to the requester and is the certificate to be used.

There are two common types of validation:

Domain validation (DV)

This is the simplest and most basic form of checking. Basically the CA tries to verify the requester has some form of control over the domain. This may be something like

Put a file on the webserver at /my/check.html containing the string “wowzer”
Edit your DNS so you have a TXT record “check_this” with the string “foobar”
Read the email sent to webmaster and click on the link in it.

It’s important to note that this doesn’t ask anything about the organisation behind the request, just whether it has some control over the domain. A phishing organisation could register the DNS name sweharr1s.org and request a cert for that.

A DV site typically shows up in browsers with a green padlock, but not much else.

The advantage of DV certificates is that the creation and signing can be done very cheaply, or even automatically (which is how Letsencrypt works).

Extended validation (EV)

EV certs take a deeper approach to validating the request; the CA also check the owner claims. So, for example, I might be able to get a DV cert for chas3.com, but I would not be able to get an EV cert saying “this is owned by JPMorgan Chase”. An EV cert is meant to help reduce phishing attacks by helping the web user spot if an EV cert is in use.

In a web browser the owner of an EV is typically presented:

When you compare that to my DV cert you can see that we both have the green padlock, showing the certs are good, but the Chase entry has more details about the owner next to the padlock.

EV certification takes more resources from the CA (typically people are involved and it’s a multi-day affair and may require a registered legal entity (e.g. a registered company name) in your country.

The idea behind EV is that you can now get a level of trust of the identity of the organisation behind a web site, which can be very important for a site such as a bank. “JPMorgan Chase and Co run this site and it is called www.chase.com” is a stronger message than “this site is called www.chas3.com”.

Organisation Validated (OV)

This is somewhat of a middle layer between DV and EV. In this setup the owner of the site is still checked. Originally this was meant to be the “higher quality” certificate but there were no strict rules around how this validation was performed (which is why the CA/Browser Forum - see later - introduced EV). That means there’s a level of assurance in the owner of the domain, but it’s not necessarily as strong as that presented by an EV cert.

OV certificates include organisational details inside the certificate, but these are rarely presented directly to the user by default. Indeed, the user may need to go digging to distinguish between DV and OV, so I’m not sure if these provide any real benefit any more.

How does your browser know the certificate is valid?

This is where the CAs signature comes in. You can use the CAs public to verify the signature is accurate (calculate the signature yourself and then use the public key to decrypt the one they provided; do they match?).

The problem is where to get the CAs public key from?

To solve this your operating system and web browser come with a set of CA keys pre-installed. For example, the Chromium browser on my Debian desktop shows a lot of CAs, including:

The sheer number of CAs can cause a problem; do you trust them all? Inherently you do, even if you don’t know it! Are you sure some random CA won’t sign a certificate for your site? Or allow some bad person pretend to be your bank? There’s a CA/Browser Forum that tries to keep on top of this which, combined with Certificate Transparency, tries to keep on top of the problem but it’s not clear if there is a good solution.

One advantage of this model, though, is that an enterprise can create their own “internal” CA; they just need to add the CA’s public key to their corporate image (or deploy it using Microsoft group policies, or similar) and their own internal created SSL certificates are trusted, which can enable internal SSL encryption to happen without needing an external CA.

What is certificate chaining

Typically the CA doesn’t sign requests with the “root” key that the browser knows about. It’s a lot of work to get new keys distributed (it’s not just browsers but also other software, such as java, that is impacted) so these keys are long lived (20 or 30 years, in some cases). If they were stolen then there would be potential for false signings for a long time. So, instead, the root key is locked in a vault and inaccessible. Not so useful. Instead an intermediate certificate (that has been signed by the root cert) is used to do the signing. These may be shorter lived (say 5 years or less).

The problem, now, is in telling the browser about this. So instead of the server just presenting the signed certificate from the CA it sends both that cert and the intermediate’s cert public key.

The browser can check the intermediate cert is good because it knows about the root cert. Once it trusts the intermediate it can then use this to check and trust the web servers certificate.

So, for example, we can see the chain for www.google.com

Here the website certificate has been signed by a Google CA, which has been signed by GeoTrust… and that is in the browser/OS certificate store and is trust. So the browser can verify the site is correct.

Certificate chaining mistakes are common in new SSL deployments, so it’s definitely worth verifying you get this correct!

What happens when a certificate expires?

Essentially nothing happens. However web browsers will start to complain. As part of their validation process they check the “not before” and “not after” dates inside the CA signed certificate and compares them to the local time. Once the certificate has expired then the browser will no longer trust it and will give the user a nasty scary looking screen.

If the user decided to click on through then they can still get to your site, and it will still be encrypted… but don’t do this. Don’t encourage users to ignore these warnings. Make sure your cert is renewed in time!

What happens when a certificate is lost stolen?

It doesn’t matter if the signed cert is stolen; all that information is public. The important part is the private key that we generated all the way back at the beginning of this post while creating the CSR. This private key is what is used to prove communication is to the server, so if someone steals that then they can decrypt your data, or even impersonate you.

There are two ways a certificate can be revoked, and these are under the control of the CA:

Certificate Revocation List (CRL). In this version the CA posts a complete list of revoked certificates. The CRL URI is in a field in the signed certificate so the client knows where to look.
e.g.
```
        X509v3 CRL Distribution Points:

            Full Name:
              URI:http://sr.symcb.com/sr.crl
```
The client must download this list and then check for the serial number of the certificate to see if it has been revoked. The Symantec list mentioned here is currently 167K in size and contains 4700 entries so clients using this approach may cache data. (Of course expired certs may be removed from the CRL, helping to keep the size down).
Online Certificate Status Protocol (OCSP). This version is more API oriented; rather than downloading a complete CRL the web browser can ask specifically for the state of the certificate. Again the URL is present in the signed certificate:
```
        Authority Information Access:
            OCSP - URI:http://sr.symcd.com
```
Entries typically have a validity time associated with them, to tell the browser how long to cache the results for.
OCSP failures have been the cause of sites becoming unreachable and have some privacy concerns, so a version called OCSP stapling, where the web server presents a signed OCSP response. Apache documentation claims this doesn’t work properly for intermediate certs, but standard OCSP caching may help, here.

Unfortunately, revocation checking isn’t necessarily reliable nor as well performed as it should be. For example, Chrome doesn’t do OCSP at all, and claims to have its own method. It’s purely a function of the client and non-browser clients rarely perform all the validation (or, indeed, any validation) on the certificates.

Summary

This was meant to be a brief introduction to some of the features of an SSL certificate, but it grew a little large! There’s a lot of features available (and I didn’t even talk about client certs, mutual TLS, code signing certs…) and it can get a little complicated.

Fortunately modern tools such as Letsencrypt handle most of this for you (it can handle the key generation, the CSR, the Domain Validation, the chaining and the rotation when certificates are due to expire… magic!). In an enterprise world, companies such as Venafi can act as an interface to hide the complexity, and integrate with your CA of choice.

So, hopefully, most people won’t need to deal with these messy internals so much. But knowing about SSL certs and how they work for assigning identity to a machine will help understand their limitations and what considerations need to in any security model relying on them.

Data At Rest Encryption (DARE)

Sun, 04 Jun 2017 16:24:09 -0400

I’ve previous written about encryption and hashing and why things like customer passwords should never be encrypted.

Sometimes, though, you need encryption because you need to get the raw data back.

Now you can apply encryption at different layers. Some are easy; some are hard. What you need to be aware of, though, is what they protect against. There is no one-size-fits-all solution

A standard app

In a common scenario we may have an application that writes data to a database; that database persists data to disk.

Device encryption

Here we’re looking at things like encrypting the whole hard disk. This could be done via software (e.g BitLocker on Windows, LUKS on Linux) or via hardware (self encrypting drives - SED). The idea is that the storage mechanism itself is encrypted, and the data is decrypted when read. The key used to enable to encrypt/decrypt cycle is typically entered at boot time; without the right key then the data is unreadable. For a laptop you would typically need to enter the password at power up; for SED devices the key may be negotiated via the hardware BIOS/TPM.

This helps prevent against physical theft; eg if someone steals your laptop then they can’t read your data without knowing the unlock password. If someone steals an SED device from your datacenter (or a site engineer accidentally walks out with one after performing a replacement) then the device won’t be able to talk to the motherboard to get the unlock key.

This is the sort of encryption used on iPhones and newer Android devices, and it’s important for the same reason that companies use BitLocker on laptops; if the device is lost or stolen then the contents can be unreadable.

Once the storage device is “unlocked” then reads and writes are encrypted transparently to the OS and applications. They don’t need to care about it.

What it doesn’t protect against, though, is against data access once the device has been unlocked. So an SA on your Linux server will be able to read all the data (the device decrypts it for them!), for example.

This layer of encryption is, essentially, transparent to higher levels of the stack which makes it nice to implement, especially on mobile devices. Everything that runs on the machine pretty much runs unchanged.

So be aware of the limitations: device level encryption can protect against loss of the device but can not protect against attacks at the OS or higher.

Database encryption

So we can go up a level; we don’t want our SAs to be able to read our data. Let’s encrypt at the database level, instead. The common term for this is “Transparent Data Encryption” (TDE). Oracle, Microsoft and MySQL are examples of this.

Now the database does the encryption/decryption. Your SA trying to read the datafiles will just see rubbish, but the application will be able to run unchanged. We’ve protected against the SA.

What it doesn’t protect against, though, is the DBA. The DBA can connect to the database and read anything. It also doesn’t protect against the SA trying to get hold of the encryption keys or gain DBA privileges (switch to the Oracle group?)

This level is effectively transparent to the database users (hence the TDE terminology), which makes it nice to implement (no application code change is necessary).

So be aware of the limitations: database level encryption can protect from disk loss or for someone being able to read the raw data files, but it doesn’t protect against authorized privileged users of the database (DBAs) or people who can become privileged users.

Application level encryption (ALE)

So now we get into the application itself; the app is modified to encrypt the data. You might use a tools from companies such as Vormetric or Voltage, which can be configured to talk to a HSM to get encryption keys.

At this stage the application can decide which columns to encrypt. You might want to encrypt a customer’s name and address, but not necessarily the date of a purchase.

ALE is the most intrusive into an application because the app needs to be modified to perform the encryption.

Depending on the type of data you may want to use Format Preserving Encryption so the data looks in the same format as the original (e.g. credit card numbers), which will allow some applications to continue to work on the encrypted data, rather than needing to decrypt it every time. (This is a good use case, when storing data in the public cloud; if your app can work on FPE data then the risk of exposure by running it in the cloud is a lot smaller than having to decrypt it each time).

Now neither the SA nor the DBA have direct access to the data, although they still may be able to abuse privileges to get hold of the encryption key or authentication tokens to talk to the HSM.

Of course your application may still need to be able to decrypt the data (how can you send an order if you can’t decrypt the name and address?), which means that you are still at risk from rogue developers or errors at the application layer, but you may be protected from a SQL injection type attack because the resulting dataset would be encrypted

What about cloud storage?

You need to think of “who has access? who can attack?”. So let’s say you’re storing stuff in Amazon S3 and decide to use server side encryption. Accidentally you allow the buckets to be read by the world, but without decryption. So you’ve protected against “raw data theft” (compare to disk leaving the data center). But you haven’t protected against theft by “authorized” users, because the S3 API will do the decryption for you. You’re also vulnerable to admins of the Amazon account who can modify the rules…

Similarly if you use EBS attached to the S3 instance and encrypt it with LUKS then you may be safe from the admins, but the SAs of the VMs can still access the data. Of course if you’re doing hands off delivery and don’t allow anyone to login then there may not be any SAs :-)

Summary

I like to look at technology as a stack of things sitting on top of each other.

So your app sits on top of a database, which sits on top of an OS, which sits on top of a machine.

Encryption at different layers protects against attacks at that level of the stack; SEDs can protect against “machine” attacks, but doesn’t protect against higher levels of the stack.

In a modern deployment you should consider device encryption as the lowest level requirement, even inside a datacenter. Disks do walk, even with the best of processes. But just because you have this low level encryption it does not mean you have “solved” the encryption problem. You need to look at your attack surface and pick a level that’s appropriate.

Sorry, developers, just because you have an encrypted disk or database TDE doesn’t mean you can avoid doing ALE!

Stuff changes; don't take things on faith, get the facts

Sun, 28 May 2017 16:58:07 -0400

A number of years back I saw a mail thread around Java performance. One person made the claim that Java was slow; if you wanted performant apps then write in a different language. This matched my experience, so I felt a little confirmation bias. However the reply really made me think; this may have been true 10 years ago (and, indeed, it’d been over 10 years since I’d done any real Java) but modern techniques meant that Java was perfectly fast. The kicker was that, instead of believing rumour and word-of-mouth ancient results, do the tests yourself; measure and get the facts.

My home server disks

My home machine has mutated a few times over the years, and suffers some technical debt as a result.

Currently the disks are connected to a couple of SAS9211-8i controllers. In theory these are PCIe 8x controllers but my motherboard has a 16x slot and a 4x slot, so one card isn’t being used at full potential. I’m using the controllers in JBOD mode, with Linux mdraid to create the various RAID options. I prefer this because it means I’m not tied to specific hardware and could migrate the disks to another setup, entirely, and the kernel should autodetect and bring up the RAID automatically.

This means the setup is currently on a CentOS 6 based system with the cards and disks distributed over the two cards:

16x slot:
- 2 Crucial CT512MX1 SSDs in a RAID 1
- 4 Seagate ST2000 in a RAID 10
4x slot:
- 8 Seagate ST4000 in a RAID 6

Now I’d been taking it as an article of faith that the SSDs would be fastest, with the RAID 10 being better than the RAID 6 for writes, and the RAID 6 being better for reads.

When I built this I ran some hdparm -t tests on the RAID volumes and was surprised how well the RAID 6 performed

The two SSDs  are in a RAID 1  and get 375 MB/sec
The 4*2Tbytes are in a RAID 10 and get 267 MB/sec
The 8*4Tbytes are in a RAID 6  and get 532 MB/sec

This made me wonder how well they’d work in a more targeted test. Since EPEL has bonnie++ in the repository I chose that. For each of the RAID volumes I picked an existing ext3 filesystem to run the tests on. This should give some form of “real world” feel, since it wouldn’t be on newly created filesystems.

I ran each test twice.

The results (slightly edited for formatting) make interesting reading:

           ------Sequential Output------   --Sequential Input- --Random-
           -Per Chr- --Block-- -Rewrite-   -Per Chr- --Block-- --Seeks--
Machine   K/sec %CP  K/sec %CP  K/sec %CP K/sec %CP K/sec  %CP  /sec %CP
Raid6       800  92 108776  14 103820  11  3276  76 606148  23 170.3   8
Raid6       784  91 108436  13 105046  11  3373  76 606068  23 172.2   8

Raid10      753  94 175905  21  91956   9  2700  76 227793  10 409.2  15
Raid10      745  91 170551  20  93408   9  3209  78 229931  10 531.6   7

Raid1 SSD   786  96 182078  21  97995  10  3685  84 236123  10 460.2  17
Raid1 SSD   796  95 173306  20  98546  10  3012  75 233678  11 470.9  16

What seems clear is that for reading the ability to distribute load over 8 disks gives a clear speed advantage when reading data blocks, but at a CPU cost. Sequential character reads don’t make a lot of difference, so we may be seeing limiting factors of the kernel and CPU and memory, rather than disk I/O here. RAID 6 definitely loses for random seeks, though!

More interesting are the writes. RAID 6 is definitely slower, but surprisingly the SSD wasn’t noticeably faster than the spinning 2TB disks. These disks are meant to be able to do 500MB/s, but I appear to get real-world speeds of 180MB/s, only a little faster than the 2TB Seagates (and those values seem reasonable, looking at this) chart.

And why are the rewrites tipped the other way?

At this point I’d almost consider moving all my data on the RAID 6 because I think that’d give the most balanced performance! However, performance isn’t the only design criteria; data risk (if a 4TB disk failed would the array survive long enough to rebuild the lost disk?) and data separation; I could turn off the RAID 6 and still keep 95% of my functionality, just long term storage wouldn’t be available, so I could run on a smaller machine if this motherboard died.

Summary

This “get the facts” can apply to many things. It becomes tempting to measure everything and try to apply change based on those numbers. But we can see, from this simple disk speed test, that numbers may not be so clear and what you measure can give you different results. If I only measure “block read” speed then I would just have everything on the RAID 6 disks; that’s so much quicker! But if I care about writing then maybe the RAID 10. Would I even care about the SSDs? And there are other factors (resiliency, recovery) to take into account. Overly optimising for one factor isn’t always the best idea, either!

Technology changes; configurations can have impacts; optimisations (JIT bytecode compilers) can add a whole new dimension. What was an article of faith 10 years ago may now hinder you and cause you to build slower more complicated solutions.

If possible you should try out various options early in the design process and pick the one that gives you best results for your use case. You need the facts in order to make the best decision. But don’t over-optimise at the cost of other factors. Decisions are not made in a vacuum.

Bottlenecks and SPOFs

Sun, 21 May 2017 11:02:58 -0400

If you’ve ever built any enterprise level system you’ll be aware of the needs of performance and resiliency. You may do performance testing on your application; you may have a backup server in a second data center; you may even do regular “Disaster Recovery tests”.

And yet, despite all these efforts, your application fails in unexpected ways or isn’t as resilient as you planned. Your primary server dies, the DR server doesn’t work properly. Why is this?

Performance tuning and hidden bottlenecks

A few weeks back I upgraded my home internet connection to gigabit speeds. The next day I found that my primary VM felt slow; commands were taking a noticeable amount of time to run when they shouldn’t (eg a simple ls). Investigation showed that my machine (a quad core i5) was maxed out, and a single process rclone was responsible. My offsite backup was running (copying data to Amazon), and I encrypt my data while sending it. By removing the network bottleneck I was now hitting a CPU performance bottleneck.

Now many applications have unexpected hidden bottlenecks as well. A common problem is when a service switches from a production data center to a secondary; this second data center may have higher network latency to their customers and this can impact performance in odd ways (using perl DBD::Oracle do a select of a million rows from a database 10ms away vs 30ms away and spot the difference!). Processing that may take most of a night may now fail to complete before the next work day; reporting deadlines could be missed, and this could have regulatory implications and fines in some industries.

This means that having a “DR” plan may not be sufficient. You should also have a Sustained Resiliency plan; don’t just verify you can shutdown your production database and spin it up in DR; keep it up and running for a week or two to verify that downstream dependent applications keep working and can meet their end-of-day, end-of-week deadlines.

Single Points of Failures

As part of our resilient design we also make choices on what services can be made highly available (e.g. a cluster) vs redundant (e.g. warm standby environment in a secondary location). We may also accept a single point of failure, based on risk factors and cost.

For example, at home if my router died then I could replace it with an openWRT based system, or even a desktop with 2 ethernet cards. It wouldn’t be as performant, but it would work for a temporary basis. Similarly if my main machine died I have a spare that I could get working (it would be messy and the case wouldn’t close…). But… I only have Verizon FIOS. If that dies then I’m off the net (my friend hit this problem recently; his ONT failed). Now I could also get Cablevision as a secondary internet provider, but this isn’t worth the monthly cost for me. So I live with the SPOF. The server this site is served from, though, does have a copy at a second provider - linode and Panix.

Another thing that may show up in a real DR scenario (as opposed to planned ones) is an unexpected dependency. Many DR tests are very fake in nature; they have a planned sequence (close down production database, sync to DR, bring up DR database, test application, switch back again). This is all clean and designed to minimise disruption to the production environment, but it’s not really representative of a real failure mode. What happens if there’s a datacenter outage? This isn’t planned; you won’t be working with a clean replicated database, you may have data-loss or require a database recovery which can impact recovery times.

Or you might find you have an unexpected SPOF dependency; is your DNS managed from the primary datacenter so you can’t modify service entries to point to your secondary? Are all writes sent to a server in the primary location? Is your authentication service hosted there?

Amazon hit this in early 2017; they found they had a massive dependency on S3 and this dependency impacted not only many many other applications but also their own status site! They couldn’t update the status page to tell people about the outage because it depended on S3. Oops!

Conclusion

You might think that modern 12-factor apps might be more immune to these failures, but you’d be wrong. You still need to think about the underlying infrastructure (“the cloud is just someone else’s computer”); deploy to multiple regions, ensure your data is properly replicated, have a sustained resiliency program, and test worst case scenarios. Even if your original architecture was good, implementations mutate and grow beyond the initial design criteria. What may have worked 2 years ago may not work now.

Be aware of hidden SPOFs (could someone DDoS your DNS servers? What would happen if your domain was hijacked?

The cloud doesn’t solve your HA/DR/SR requirements; it just provides tools to assist in building resilient solutions.

Building a home router

Sun, 07 May 2017 13:50:18 -0400

WARNING: technical content ahead! There’s also a tonne of config files, which make this page look longer than it really is, but hopefully they’ll help other people who want to do similar work.

For many years I’ve been using variations of the Linksys WRT54G. I first switched to this router when freeware ROMs became available; I’ve used DD-WRT, Tomato, OpenWRT and others. I love the flexibility it gives me. My current router is a TP-Link TL1043ND with OpenWRT. It works well, and the flexibility means it’s also configured as my OpenVPN endpoint, HEnet IPv6 tunnel endpoint, dual SID WiFi access point

However I have some problems with this setup:

Upgrades are painful. If there’s a bug in some core code (eg openvpn) then the only real fix is to find the latest software and reflash; you can’t just upgrade the individual packages, even if they were installed as extras. This has meant that my router may technically be vulnerable to known issues.
Network speed. This week I upgraded my FIOS connection to gigabit speeds; the TP-Link maxes out around 175Mbit/s and so is now the bottleneck.
It’s not exactly performant; the OpenVPN speed is horribly slow. This isn’t really a problem since I just use it as an admin channel between my home network and my internet hosts, but it’d be nice to resolve.

Since I needed a newer faster router I decided to try and fix these problems at the same time. This, basically, meant building a real PC with a real OS and configuring it as a router. I could have looked at something like pfsense but that’s a different OS to manage. I didn’t particularly need a pretty GUI, either.

So I went with a CentOS 7 build.

The hardware

The nice thing about these WRT routers is they are cheap. Of course there are tradeoffs (performance). I wanted something in the same ballpark and settled on a PC Engines APU2 board. They only had the 2Gb RAM model in stock when I ordered, but that should be sufficient:

System Board: $102.00
        Case: $ 10.00
   16Gb SATA: $ 14.40
   WiFi card: $ 19.00
    Antennas: $  4.20
      Cables: $  3.00
    Shipping: $ 14.30
              =======
              $166.90

That’s in the right ballpark. It’s a fanless system, so quiet, and just runs from a DC power adapter (which I had available). The one downside is that the WiFi card can only work on 2.4 or 5Ghz, not both at the same time, but otherwise this is just about right.

The PC Engines team have a number of support files which allow you to generate boot USB drives to allow you install your OS of choice, including (in my case) CentOS 7. And the machine has a serial console, making it easy to remote manage. You don’t need to plug a keyboard, mouse, monitor in (indeed, there’s no VGA/HDMI output!)

The OS

I installed a pretty minimal CentOS 7 image on the machine. Why C7? Because that’s what the majority of my systems are based on. It meant I could use my standard backup, monitoring, patching, control processes. I removed NetworkManager, firewalld, unnecessary firmware. I added the EPEL repo. In particular I added

hostapd to be the Wifi AP manager
bridge-utils to manage the bridging
iw and wireless-tools to manage the WiFi
openvpn for the VPN endpoint
iptables-service to allow me to save/restore IPtable configurations
dhcp and dnsmasq to serve guest-wifi addresses
radvd for IPv6 routing

Basic network layout

I want to have 3 networks:

WAN
LAN
Guest-net

To make rules easier to follow I created 3 bridges (br-wan, br-lan, br-guest). I was then able to associate physical and logical devices to this.

So, for example, in /etc/sysconfig/network-scripts I could configure the WAN port:

% cat ifcfg-br-wan
DEVICE=br-wan
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
PEERDNS=no

% cat ifcfg-enp2s0 
DEVICE=enp2s0
TYPE=Ethernet
ONBOOT=yes
BRIDGE="br-wan"
NM_CONTROLLED="no"

This puts the second ethernet port onto the WAN bridge and configures it for DHCP, which is what my ISP uses. I plug in the cable and…

% ip -4 addr show dev br-wan
13: br-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    inet 173.54.110.200/24 brd 173.54.110.255 scope global dynamic br-wan
       valid_lft 4808sec preferred_lft 4808sec

Perfect, the machine is now on the internet… and succeptible to being attacked. Indeed, I saw one ssh scan happen while I was building this; I should have locked down traffic before I started, but I felt the risk was low (only 1 account has password access and that has a hard password).

The LAN side can be similarly configured, but with a static IP:

$ cat ifcfg-br-lan
DEVICE=br-lan
TYPE=Bridge
ONBOOT=yes
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_AUTOCONF=no
IPV6ADDR=2001:470:1f07:dc4::1/64
BOOTPROTO=none
IPADDR=10.0.0.1
NETMASK=255.255.255.0
BROADCAST=10.0.0.255
NETWORK=10.0.0.0

$ cat ifcfg-enp1s0
DEVICE=enp1s0
TYPE=Ethernet
ONBOOT=yes
BRIDGE="br-lan"
NM_CONTROLLED="no"

Note I’ve given this an IPv6 address as well.

And br-guest is similarly static, but (at this point) with no ethernet port assigned:

$ cat ifcfg-br-guest 
DEVICE=br-guest
TYPE=Bridge
ONBOOT=yes
BOOTPROTO=static
IPADDR=10.100.100.1
NETMASK=255.255.255.0
DEFROUTE=no
IPV4_FAILURE_FATAL=no
IPV6INIT=no

The observant will have noticed that I haven’t mentioned WiFi - that gets configured next!

WiFi

The hostapd program creates the WiFi networks and controls the bridges the networks are connected to. The WiFi card in this machine can host multiple SSIDs at the same time. In my case I defined two networks (LAN and GUEST) with their own WPA passwords and associated to the relevant networks.

$ cat /etc/hostapd/hostapd.conf
ctrl_interface=/var/run/hostapd
ctrl_interface_group=wheel

macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0

driver=nl80211

interface=wlp4s0
hw_mode=g
channel=1

ssid=LANSSID
ap_isolate=1
disassoc_low_ack=1
preamble=1
wmm_enabled=1
ignore_broadcast_ssid=0
uapsd_advertisement_enabled=1
wpa_passphrase=YOURNETWORKPASSWORD
auth_algs=1
wpa=2
wpa_pairwise=CCMP
bridge=br-lan
wpa_key_mgmt=WPA-PSK
okc=0
disable_pmksa_caching=1
bssid=04:f0:21:26:d3:7e

bss=wlp4s0_1
ssid=GUESTSSID
ap_isolate=1
disassoc_low_ack=1
preamble=1
wmm_enabled=1
ignore_broadcast_ssid=0
uapsd_advertisement_enabled=1
wpa_passphrase=YOURGUESTNETPASSWORD
auth_algs=1
wpa=2
wpa_pairwise=CCMP
bridge=br-guest
wpa_key_mgmt=WPA-PSK
okc=0
disable_pmksa_caching=1
bssid=04:f0:21:26:d3:7f

This configures the wlp4s0 (WiFi card) interface but also creates a second logical interface wlp4s0_1 for the guest network.

Bridge results

We can verify what interface is on what network quite simply.

% brctl show
bridge name     bridge id               STP enabled     interfaces
br-guest        8000.04f02126d37f       no              wlp4s0_1
br-lan          8000.000db9439ccc       no              enp1s0
                                                        wlp4s0
br-wan          8000.000db9439ccd       no              enp2s0

Guestnet DHCP and DNS

For my LAN I have an ISC DHCP and BIND setup running on another machine; machines on the LAN network (whether cabled or wireless) will talk to them; I don’t need to worry. But guestnet users can’t reach these. So the router needs to provide these functions.

The out-of-box configuration for dnsmasq pretty much works just fine. The only change I added was no-hosts to prevent it looking up entries in the /etc/hosts file. At the moment I’m letting it use the DNS server list in /etc/resolv.conf as the server to forward to (ie my internal host) but I might reconfigure it to point to Google instead.

If I do that then I expect (untested!) the configuration to be

no-resolv
server=8.8.8.8

This would allow me to use this router’s DNSmasq as a secondary DNS for LAN usage. (Food for thought).

For DHCP I’m sticking with ISC dhcpd, with a simple configuration:

ddns-update-style  none;

option option-128 code 128 = string;
option option-129 code 129 = text;
option option-221 code 221 = text;

subnet 10.100.100.0 netmask 255.255.255.0 {
  authoritative;
  option routers      10.100.100.1;
  option subnet-mask    255.255.255.0;
  option domain-name    "spuddy.org";
  option domain-name-servers  10.100.100.1;
  default-lease-time 3600;
  max-lease-time 3600;
  range dynamic-bootp 10.100.100.10 10.100.100.200;
}

Basically, serve out addresses with a default route and DNS server. Pretty simple!

IPv6

Setting up an IPv6 tunnel

If have a HEnet tunnel for my home IP range. This is pretty easy to set up with CentOS:

DEVICE=sit1
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6TUNNELIPV4=IP_ADDRESS_OF_REMOTE_SERVER
IPV6ADDR=YOUR_ASSIGNED_IPv6 ENDPOINT

And that’s pretty much it. As long as your IPv4 address is assigned at HE.net then the tunnel comes up and runs.

You can do this with a dynamic script that probes your external IP address and then calls the tunnelbroker API:

PASS=YOUR_PASSWORD
USERID=YOUR_USERNAME
GTUNID=YOUR_TUNNEL_ID
HOST=https://$USERID:$PASS@ipv4.tunnelbroker.net

newip=$1

# Check to see what the end point currently is
x=$(wget -q --no-check-certificate -O - $HOST/tunnelInfo.php?tid=$GTUNID | sed -n 's/^.*<clientv4>\(.*\)<.*$/\1/p')

if [ "x$x" == "x$newip" ]
then
  echo Tunnel IP is already correct
  exit
fi

wget -q --no-check-certificate -O - "$HOST/nic/update?hostname=$GTUNID&myip=$newip"

With a script similar to this even if your IP address changes then you can make sure the tunnel endpoint is updated and your IPv6 connection will restore.

(Hmm, I run that script from a host on my LAN; I wonder if I could make it a triggered event; see the “reflection” section below on this)

Setting up `radvd`

In order for the rest of your network to see the IPv6 world we use radvd. This allows hosts to auto-discover their IPv6 network and default route.

We just configure it on the LAN network:

$ cat /etc/radvd.conf 
interface br-lan
{
        AdvSendAdvert on;
        MinRtrAdvInterval 30;
        MaxRtrAdvInterval 100;
        prefix YOUR_IP_V6_NETWORK/64
        {
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr off;
        };
};

In my case the IPv6 network is that from HE.net; 2001:470:1f07:dc4::/64 as matches the IPv6 address defined in ifcfg-br-lan

Firewalls and routing

Now we have the plumbing out of the way we can start to make the device act like a router.

Let’s make the kernel forward traffic first:

$ cat /etc/sysctl.d/forward.conf 
net.ipv4.ip_forward = 1
net.ipv4.conf.all.forwarding = 1
net.ipv6.conf.all.forwarding = 1

At this point your LAN and guest networks can see each other, and you should be able to see your external (ISP provided) IP address. But not much else on IPv4. You should have full internet access via IPv6… and the rest of the internet can see you, as well!

iptables is used to configure both NAT and firewall rules; ip6tables is used to configure IPv6 firewall rules. We don’t need NAT on a routed IPv6 network.

There’s two ways of doing this; you can run the iptables command to add rules, or you can edit /etc/sysconfig/iptables files directly. If you use the commands then run service iptables save to update the config file so the rules persist after a reboot. If you edit the file directory then run service iptables reload to force them to be reloaded.

I tend to do edit the file directly!

IPv4 rules

when looking at standard rules (INPUT/OUTPUT/FORWARD) we have to look at them from the perspective of the router. INPUT rules are those that impact packets that are targetted at the router; OUTPUT rules are those that originate from the router; FORWARD rules are those that pass through the router. This gets important for NAT where the packet targets the router’s external IP address.

First we need to allow incoming traffic that corresponds to outgoing connections (if you reach out on port 80 then you want to allow the remote end to reply!). We’ll add two basic rules to the 3 main filter channels. From the command line this would look like:

iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate INVALID -j DROP
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate INVALID -j DROP

You’ll see similar entries in the *filter section of the config file.

Now we want to allow some specifics. The LAN needs to see the WAN, for example. We can add some more INPUT rules:

-A INPUT -i lo -j ACCEPT
-A INPUT -i br-lan -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -p tcp -m tcp --dport 12345 -j ACCEPT
-A INPUT -i br-guest -j guest_reflection
-A INPUT -j REJECT --reject-with icmp-port-unreachable

You can see, now, why I created these bridge devices. It becomes pretty clear what networks rules apply to! We’ll talk about the “reflection” rule later.

The --dport 12345 is where I run my OpenVPN server. This allows the remote machines to access it.

Forwarding is similar:

-A FORWARD -i br-lan -j ACCEPT
-A FORWARD -i tun0 -j ACCEPT
-A FORWARD -i br-wan -m conntrack --ctstate DNAT -j ACCEPT
-A FORWARD -i br-guest -o br-wan -j ACCEPT
-A FORWARD -i br-guest -j guest_reflection
-A FORWARD -j REJECT --reject-with icmp-port-unreachable

The br-wan rule is important when we get to port forwarding. We also only allow the guest network to see the WAN; it can no longer reach your LAN machines.

The output chain doesn’t have any special requirements; we allow any packet originating from the router to be sent out.

So far so good. But you still can’t see the outside world from the LAN. So now we get to the NAT table. This is configured with iptables -t nat, or by entries in the *nat section of the config file.

With NAT we have INPUT, OUTPUT, PREROUTING and POSTROUTING sections.

The first step is (remember, if from the command line to add -t nat)

-A POSTROUTING -o br-wan -j MASQUERADE

This is the magic that allows outgoing communication. Every machine on your LAN and guest networks should now be able to see servers on the internet.

Many people (me included!) also want to allow some services internally to be visible externally. For example, I run my own IMAPS server which I have my phone configured to talk to. We need to allow this traffic through:

-A PREROUTING -i br-wan -p tcp -m tcp --dport 12346 -j DNAT --to-destination 10.0.0.137:993

This tells the router that anything arriving from the WAN targeting port 12346 should be redirected to the internal server 10.0.0.137 on the IMAPS port.

You can define a series of ports; eg for bittorrent I have

-A PREROUTING -i br-wan -p tcp -m tcp --dport 6881:6999 -j DNAT --to-destination 10.0.0.137:6881-6999

These work in conjunction with the earlier FOWARDING DNAT rule I mentioned to let the traffic through.

IPv6 rules

Because there’s no NAT needed here we can have simpler rules. I just allow my external servers to access the internal machines, and nothing else. That makes for a very trivial set of rules in /etc/sysconfig/ip6tables

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -i lo -j ACCEPT
-A INPUT -i br-lan -j ACCEPT
-A INPUT -p ipv6-icmp -j ACCEPT
-A INPUT -s ip:6:of::HOST1/128 -j ACCEPT
-A INPUT -s ip:6:of::HOST2/128 -j ACCEPT
-A INPUT -s ip:6:of::HOST3/128 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp6-adm-prohibited
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -i br-lan -j ACCEPT
-A FORWARD -s ip:6:of::HOST1/128 -j ACCEPT
-A FORWARD -s ip:6:of::HOST2/128 -j ACCEPT
-A FORWARD -s ip:6:of::HOST3/128 -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp6-adm-prohibited
-A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -m conntrack --ctstate INVALID -j DROP

As with IPv4, the INPUT rules allow those hosts to access the router directly, the FORWARD rules allow it to access machines behind the router on my LAN.

Reflection

So far we now have a working router. We can reach the internet and allow the internet to reach into our network. But there’s one tiny gotcha. Which is an odd edge-case that you may not care about…

Access from your internal network (LAN, guestnet) to your external IP address will not work! That’s because the traffic to those external ports doesn’t come in from br-wan and so isn’t rewritten. Now OpenWRT creates a series of additional rules called “reflection” rules. They take the packet and rewrite it so it looks like it comes from the router, and then sends it back in to the LAN. It’s a kludge, but it works.

In the NAT section I added two rules:

-A PREROUTING -s 10.0.0.0/8 -p tcp -m tcp -j reflection_pre
-A POSTROUTING -s 10.0.0.0/8 -p tcp -m tcp -j reflection_post

Now this allows us to reflect traffic originating from any internal IP address (LAN, guestnet) by creating rules in the new chains.

The problem is that these rules need to refer to the router’s external IP address, and this can change. So I added a hook into dhclient’s configuration, which gets run whenever an IP address is associated:

$ cat /etc/dhcp/dhclient.d/reflect.sh
#!/bin/bash

I="/sbin/iptables"
N="$I -t nat"

get_ip()
{
  /sbin/ip -4 addr show dev $1 | sed -n 's/^\s*inet \(.*\)\/.*/\1/p'
}

LAN_ADDR=$(get_ip br-lan)
LAN=${LAN_ADDR%.*}.0/24
GUEST_ADDR=$(get_ip br-guest)
GUEST=${GUEST_ADDR%.*}.0/24
ALL_LAN=${LAN_ADDR%.*}.0/8

reflect()
{
  $I -A guest_reflection -s $ALL_LAN -d $1/32 -p tcp -m tcp --dport $(echo $2 | tr - :) -j ACCEPT

  $N -A reflection_post -s $ALL_LAN -d $1/32 -p tcp -m tcp --dport $(echo $2 | tr - :) -j SNAT --to-source $LAN_ADDR
  $N -A reflection_pre -s $ALL_LAN -d $new_ip_address/32 -p tcp -m tcp --dport $(echo $3 | tr - :) -j DNAT --to-destination $ip:$i_port
}

reflect_restore()
{
  if [ "$interface" = "br-wan" -a -z "$new_ip_address" ]
  then
    $N -N reflection_post 2> /dev/null
    $N -N reflection_pre  2> /dev/null
    $N -F reflection_post
    $N -F reflection_pre
    $I -N guest_reflection 2> /dev/null
    $I -F guest_reflection
  fi
}

reflect_config()
{
  if [ "$interface" = "br-wan" -a -n "$new_ip_address" -a "$new_ip_address" != "$old_ip_address" ]
  then
    $N -N reflection_post 2> /dev/null
    $N -N reflection_pre  2> /dev/null
    $N -F reflection_post
    $N -F reflection_pre
    $I -N guest_reflection 2> /dev/null
    $I -F guest_reflection

    $I -L -t nat -n | sed -n 's/^DNAT.*tcp dpts*:\(.*\) to:\(.*\):\(.*\)/\1 \2 \3/p' | while read e_port ip i_port
    do
      reflect $ip $i_port $e_port
    done
  fi
  $N -A reflection_post  -j RETURN
  $N -A reflection_pre   -j RETURN
  $I -A guest_reflection -j RETURN
}

Now what this does is look at the existing port forwarding rules you’ve defined and then builds the necessary forwarding and NATting rules. So, for example, for the IMAPS port I can now see:

# iptables -n -L guest_reflection -v | grep 993
    0     0 ACCEPT     tcp  --  *      *       10.0.0.0/8           10.0.0.137           tcp dpt:993

# iptables -t nat -n -L reflection_pre -v | grep 993  
    0     0 DNAT       tcp  --  *      *       10.0.0.0/8           173.54.110.200       tcp dpt:12346 to:10.0.0.137:993

# iptables -t nat -n -L reflection_post -v | grep 993
    0     0 SNAT       tcp  --  *      *       10.0.0.0/8           10.0.0.137           tcp dpt:993 to:10.0.0.1

Final configuration:

# cat /etc/sysconfig/iptables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:guest_reflection - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -i lo -j ACCEPT
-A INPUT -i br-lan -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -p tcp -m tcp --dport 12345 -j ACCEPT
-A INPUT -i br-guest -j guest_reflection
-A INPUT -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -i br-lan -j ACCEPT
-A FORWARD -i tun0 -j ACCEPT
-A FORWARD -i br-wan -m conntrack --ctstate DNAT -j ACCEPT
-A FORWARD -i br-guest -o br-wan -j ACCEPT
-A FORWARD -i br-guest -j guest_reflection
-A FORWARD -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -m conntrack --ctstate INVALID -j DROP
COMMIT
*nat
:PREROUTING ACCEPT [4:2876]
:INPUT ACCEPT [1:60]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:reflection_post - [0:0]
:reflection_pre - [0:0]
-A PREROUTING -i br-wan -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.0.0.2:80
-A PREROUTING -i br-wan -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.0.0.2:443
-A PREROUTING -i br-wan -p tcp -m tcp --dport 12346 -j DNAT --to-destination 10.0.0.137:993
-A PREROUTING -i br-wan -p tcp -m tcp --dport 6881:6999 -j DNAT --to-destination 10.0.0.137:6881-6999
-A PREROUTING -s 10.0.0.0/8 -p tcp -m tcp -j reflection_pre
-A POSTROUTING -s 10.0.0.0/8 -p tcp -m tcp -j reflection_post
-A POSTROUTING -o br-wan -j MASQUERADE
COMMIT

# cat /etc/sysconfig/ip6tables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -i lo -j ACCEPT
-A INPUT -i br-lan -j ACCEPT
-A INPUT -p ipv6-icmp -j ACCEPT
-A INPUT -s ip:6:of::HOST1/128 -j ACCEPT
-A INPUT -s ip:6:of::HOST2/128 -j ACCEPT
-A INPUT -s ip:6:of::HOST3/128 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp6-adm-prohibited
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -i br-lan -j ACCEPT
-A FORWARD -s ip:6:of::HOST1/128 -j ACCEPT
-A FORWARD -s ip:6:of::HOST2/128 -j ACCEPT
-A FORWARD -s ip:6:of::HOST3/128 -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp6-adm-prohibited
-A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -m conntrack --ctstate INVALID -j DROP
COMMIT

Summary

That took me around a day to build. It doesn’t have a pretty GUI, it doesn’t give me speed graphs or track usage. It’s a very basic router. Which is what I want. I could add stuff like bandwidth graphs (which might be interesting) by monitoring packets per minute. They’re “day 2” deliverables.

My “day 1” was to build a router to replace my TP-Link and that was capable of making better use of my gigabit link. And I think I’ve mostly done this. It’s actually hard to tell, because my desktop only has a 100Mbit ethernet card in it so sites like speedtest.net or fast.com just tell me the speed my desktop can handle!

For the past few years I’ve been doing a regular wget of a 100Mbyte file from http://cachefly.cachefly.net/100mb.test but this now takes around 1 second and so the startup cost and latency become a determining factor in the speed calculation. I’m getting speeds around 760Mbit/s, but would a larger file give better results!

2017-05-07 14:50:02 (95.2 MB/s) - `/dev/null' saved [104857600/104857600]
2017-05-07 15:05:02 (78.6 MB/s) - `/dev/null' saved [104857600/104857600]
2017-05-07 15:20:02 (88.1 MB/s) - `/dev/null' saved [104857600/104857600]
2017-05-07 15:35:02 (93.0 MB/s) - `/dev/null' saved [104857600/104857600]

Similarly hitting other sites may give slower results because the bandwidth across the internet could be the limiting factor.

It’s funny how local and remote infrastructure can now be the limiting factor in testing speeds; previously the link to the ISP was the bottleneck.

This showed up nicely in my morning backups; every Sunday I do a level 0 dump and then put an encrypted copy into my Amazon Cloud Drive. Last week that took almost 4 hours to upload 100G. Today it took around 30 minutes, which means I was uploading at around 480Mbit/s. And there the limiting factor might have been the CPU ‘cos the process was using 380% CPU to do the encryption, on my 4-core machine!

It’s fast enough for me, anyway ;-)

Cloud Inventory

Sun, 30 Apr 2017 16:16:58 -0400

One of the golden rules of IT security is that you need to maintain an accurate inventory of your assets. After all, if you don’t know what you have then how you can secure it? This may cover a list of physical devices (servers, routers, firewalls), virtual machines, software… An “asset” is an extremely flexible term and you need to look at it from various viewpoints to ensure you have good knowledge of your environment.

Traditionally many of the tools we use to track this stuff are based around a relatively static concept of machine persistency; if it takes months to get tin onto the datacenter floor and then it’ll stay there for 3 or more years then it makes sense to track physical assets in that way. Similarly if you build a VM and it stays around for a year and runs a known application then we can use that VM as the basis for an inventory record. We can create tools that use the OS as the core record and scan the machine (e.g. ssh in) and discover relationships (child VMs; OS version; software running; communications between servers; etc) and automatically build out the inventory. This is good standard enterprise IT management.

The cloud, though, can change this. I can spin up a server in under a minute, run my application for 2 hours, then shut it down again. Tomorrow I can spin up another copy (with a new machine identity, new IP address, etc). The machine isn’t up long enough to be scanned, and even if we could then the records would be stale and useless.

So we need to rethink how we deal with inventory in a cloud world.

IaaS

The first thing people typically think of, when considering cloud, is the IaaS model. Spin up an AWS VPC, create networks, VMs, install your app. We’re in the cloud! And this is fine. You’re basically treating the cloud as an outsourced data center, with the assets being an extension of your core environment. In this scenario your existing tools and processes may well be sufficient.

However a primary feature of an IaaS is that it can be a lot more dynamic than that. There’s an API to let you build new assets and these are available almost immediately. I could run a script that builds a new VPC, creates a three tier network architecture, installs VMs, deploys my app. Within minutes I have a full stack.

This can lead to a shadow IT problem, so it becomes very important to control IaaS services. Central IT should provide the tools and the gateway services to the cloud and not allow direct administrative access.

This can also be used to solve the “need to scan” nature of existing systems; your tooling can tag the data by requiring this to be specified up front (“I’m building 3 VMs with RedHat 7 to run application XYZ and it needs to listen on port 443 and speak to database ABC”). You have the data needed for your inventory, up front!

The same can be true for other sorts of infrastructure (networking, WAFs, firewalls, storage). If you must permit this level of administrative access to the cloud provider then ensure there’s an audit trail of activity (e.g. AWS CloudTrail) that can be used to track the API calls and maintain the inventory information, accordingly.

However we’re not quite finished… a lot of inventories are point in time snapshots; “this is what we have now”. Unfortunately a data breach may not be detected for quite a while (I’ve see 200 days thrown around a lot). Since your cloud server may have been shut down, IP address re-used for another application, and a new instance created elsewhere, we need the ability to “time travel”; maintain an audit trail to allow us to recreate the state of the inventory at whatever time we care about.

And, bonus! This also helps speed up delivery of internal IaaS services.

PaaS

In some respects an off-premise PaaS provider is probably easier to manage than an IaaS; PaaS engagements are typically more “heavy weight” than just spinning up a few VMs, and the teams using them are more likely to follow corporate processes around vendor management (especially if the team wants a VPN into the PaaS environment), and this can help ensure the application inventory is correctly maintained. At this point we can’t create an “application to host” mapping, because there is no fixed host, but we can easily see how the PaaS provider becomes an equivalent parent entity.

Of course some teams may bypass the controls (another Shadow IT issue), especially where the PaaS is directly reachable and management via https. Some of these concerns can be addressed in a similar way to SaaS, which we’ll get to in a moment.

An on-premise PaaS can be a challenge for existing processes to handle. It’s tempting to treat this similar to an off-prem solution (“my application runs on the inhouse PaaS”), but we lose some of the ability to determine impact of a breach. Common post-breach investigation processes use the inventory to determine application scope impact, but with a PaaS almost any application in the environment could be co-located on the same physical machine at the time an application was broken into. Inventory and audit trails don’t help so much here because elastic applications, especially 12 factor applications, are highly dynamic inside a PaaS environment. Yes, this also impacts off-prem PaaS, but we’re accepting those are opaque boxes; with an on-prem PaaS we have the ability to take the lid off the box… and this could hamper us!

I’d love to hear from any DFIR experts on how they handle (or plan to handle) PaaS breaches; the best I’ve heard, so far, is “comprehensive audit logs of everything that happens”). Please, feed back welcome!

SaaS

Now we come to the scariest part of the cloud, mostly because your organisation ~~may be~~ are using SaaS services without your knowledge. Wolf Goerlich claims that your average organisation uses 928 cloud services. There’s also a claim that CISOs reckon they only use 40 cloud apps. Now there may a discrepancy between “cloud app” and “cloud service” accounting for this massive gap but, even so, 928 is a lot, and most of these will be SaaS offerings.

We’re not talking about your centrally managed service (e.g. the corporate Office365, or corporate Google Apps); those we have pretty good control over and can track their use and understand them as assets.

The problem is the unmanaged services. This is because many of these SaaS offerings are just ad-hoc web sites that people use without thinking. “I want to pretty format my perl code; ah, here’s a web site; let’s cut’n’paste into that form…”

Effectively this is almost all “shadow IT”, and one that’s hard to identify.

You might wonder why that is a problem… well, that web site now has a copy of your code. Do they keep it? What terms and conditions were on that site? Many of them may claim ownership of the data submitted; has your developer now given away that code, just for the benefit of having it nicely formatted?

Even if the T&Cs are good, has the site ever been breached? How can you be sure there isn’t a bad guy on that service stealing data?

Then there’s third party repository services; developers may pull down modules or libraries from docker, or nodejs, or CPAN or… These services may not be reliable (leftpad anyone?), or have rogue code added, and can lead to non-repeatable builds.

How can you manage this if you don’t even know what is in use?

The first step is to identify what your teams are doing; what 928 services are being accessed. Then identify which of these may be considered high risk (eg T&Cs that you don’t like; poor reputation) and then start blocking them at your border. You do force people to go via proxies, don’t you?

This starts to sound like a lot of work and is likely an area worth purchasing a service to handle; these can take your proxy logs and identify who is accessing what, whether it’s mostly downloads or if there are large uploads (which could identify data exfiltration), and provide reports on your risk posture. (And this can also help with the shadow IT PaaS issue, mentioned earlier).

Ultimately you may wish to create inhouse alternatives; if you find lots of people hitting code formatting sites then create your own! Create your own code repository service that has curated library/code modules so you’re not dependent on external services. Perhaps find a service provider that does what the adhoc service does and enter into a managed agreement.

Conclusion

There’s no one size fits all solution to managing a cloud inventory. For managed services this can be done via central IT tooling (IaaS) or identified service agreements (SaaS); we need to change our technology processes to handle the more dynamic nature of the environments, but we can deal with this in various ways (e.g. dedicate a VPC to an application? Then it doesn’t matter so much about VM spinup/spindown; all VMs inside that VPC are dedicated to the app)

The bigger gap is the unmanaged SaaS footprint. The first step is identification. The second is control. This can be made easier with strong egress controls on your environment, forcing traffic to go through choke and monitoring points. You want to do this for Data Loss Prevention, malware scanning, blocking phishing sites and so on; the same infrastructure can be used to detect and limit risking SaaS usage… and to keep an inventory of what SaaS services are in use!

Persistent Applications

Sun, 23 Apr 2017 16:54:18 -0400

A while ago I wrote about some of the technology basics that can be used for data persistency. Apparently this is becoming a big issue, so I’m revisiting this from another direction.

Why does this matter?

In essence, an application is a method of changing data from one state to another; “I charge $100 to my credit card” fires off a number of applications that result in my account being debited, and the merchant being credited. Similarly when my employer pays me a million bucks (hey, it could happen!) there’s another set of changes.

At the end of the day, a large proportion of applications operate on some form of persistent data.

persistence on non-persistent HTTP.

Web developers are used to dealing with non-persistency. After all, every HTTP request is a unique connection. Developers learned to deal with this.

In the very early days it was done by the construct of hidden fields in a form. Each step in a multi-step process resulted in the web page, itself, persisting the state from the previous steps.

When cookies came along, some of this information was stored in the cookie. And then the concept of a session ID came along, with data persisted on the server and the client just identifying the session. The languages grew to help and hide this complexity (eg PHP session IDs).

When we expanded to multiple servers and load balancers the load balancer learned how to interpret the session headers and direct traffic to the same node.

Today this type of persistency over a non-persistent transport is considered normal. We know the failure modes (e.g. server reboots) and the smart developer can handle this state.

Non-persistent containers

This web development model nicely grows into the modern container world. Containers can easily have a temporary writeable overly /tmp and so your existing PHP app can maintain its state database for the lifetime of the container. We may reboot containers more frequently, though, so a smart developer might use a distributed store such as etcd; now it doesn’t matter what container processes the request, it has access to the transaction state. This small change in development actually makes life easier; the load balancer doesn’t need to understand state because any container instance is good and so the least loaded instance can be used.

Non-web apps; Lift and shift

The world isn’t so clean cut for non-web apps. A number of these apps are based inherently on the concept of a persistent infrastructure layer. This may require a code change, or else using the “machine container” pattern aka “container as a lightweight VM”. With a small amount of effort the app may be rewriteable to work with an attached persistent store, or to use an external persistence layer (database, S3 store, something else). The lift-and-shift approach gives least benefits of the container model; you might want to reconsider why you want to containerize this app.

Databases

A database is just a special case of a non-web app. We could add a persistent backend store (the equivalent of a -v flag with docker) to the container spin up, and treat it as a cluster build. We can then spin containers and the new image will reconnect to the existing backend store and act as if it was a recovering database rejoining the cluster

Summary

Not everything needs to be placed in a container; is it the right technology solution? However, containers are flexible. There’s more than one way to use them. A PaaS, such as CloudFoundry or Apprenda, pushes for the 12 factor approach of development. This gets a lot of headlines. But it’s not the only approach. An existing web app can normally be easily ported to an immutable container model (just with a writeable /tmp). A small amount of rework can allow a more traditional app to work. Or we can use the “lightweight VM” approach and persist data.

I find the “containers, containers, containers” mantra to be a little offputting. For decades I’ve maintained “use the right technology for the job”, whether that’s a Windows machines or a Linux machine or even a mainframe from the dinosaur pool (they have their uses!). Similarly, physical, virtual, container… and the type of container model; there is no one size fits all solution. Pick what is right for your use case.

Multifactor Authentication

Sun, 16 Apr 2017 15:35:49 -0400

We’re all used to using passwords as an authenticator. However, passwords have a number of problems. In particular people tend to re-use them on other sites (so if one website was broken into the password used there may also work on another site). Also passwords are susceptible to replay attacks (even if you force a password change every 90 days, there’s still a large window of time where a stolen password can be used).

To mitigate this risk, a second factor may be used.

What is multi-factor authentication.

Typically we look at authentication tokens as being classed in one of three ways. Collequially these are termed:

Something you know. Typically this is a password
Something you have. Maybe a physical token of some form
Something you are. This is form of biometric, such as a fingerprint

These are sometimes jokingly referred to as “something you’ve forgotten; something you’ve lost; something you’ve had chopped of”.

Multi-factor authentication is where 2 or more different factors are used. The “different” part is important. If, for example, we used two passwords (two “something you know” factors) then we are not any stronger than if we forced one larger password.

e.g. if password1 was “hello” and password2 was “there” then we have the same as if we only had “hellothere” as the password. Indeed, depending on the implementation, it may be weaker!

You probably already use two factor authentication in your every day life; each time you go to the ATM to withdraw cash you are presenting two factors:

Something you know (the PIN)
Something you have (the card you enter into the machine).

Two Factor Autentication (2FA)

Here you present two factors to the system. Both of these factors are checked, and if you’ve entered them correctly then you are let in.

Critically, for 2FA, if you get either factor wrong then you are not told what factor is wrong.

Weaker 2FA

This is very similar to 2FA, but the two authenticators are checked separately and you are told what one is wrong. This is weaker than 2FA because each factor can be attacked separately.

An example of this might be a web site where you are asked to enter your password. If you get your password correct then you are asked to enter a PIN, but if you get the password wrong then you are told to re-enter the password. You can keep on attacking the password until you are asked for a PIN; once you get the PIN prompt you know the password is correct.

To complicate things slightly, sites such as the AWS web console ask for passwords and PINs separately, but they always ask for the PIN whether you get the password correct or not, so this is, potentially, strong 2FA.

A common strong form of “something you have” is an RSA SecurID token.

This device generates a number based on the current time and a secret unique to the device. It’s a string One Time Password (OTP) generator, and is one of the most common “something you have” tokens. Unfortunately each company would typically require a unique RSA token and they need replacing every few years, so you could end up carrying a lot of these around. To try and solve this problem there’s now software versions of these keys (“soft token”) that may be tied to a device (e.g. using the phone device ID) and require a PIN to unlock, but they could be stolen…

One “something you have” token I like the idea of is the YubiKey FIDO U2F.

This uses public key crypto in a clever way, to allow the same key to be used on multiple sites and is really cheap. Unfortunately client side support is poor at the moment and worse, from my perspective, many organisations are locking down USB ports (for good reason) and so making use of these devices hard to introduce.

I’m not so much of a fan of biometrics (“something you are”). Fingerprints are the most common, but some voice response systems are now attempting voice recognition as a second factor, and retina scan systems also exist. The problem I have with these is that they’re hard to revoke. What happens if your fingerprint gets stolen (you leave a million copies lieing around every day!); you can’t really change it! It may be good enough to unlock your phone, but I wouldn’t use it as a single factor.

Two step verification (2SV)

2SV is very similar to 2FA in that it asks for two authenticators. In this case, though, the second authenticator is typically a OTP transmitted to you out of band. A common example is the first time you login to your bank web site from a new machine (or new browser); it might want to send you a one time number via SMS to your phone that you need to type into the website.

Superficially, this looks like 2FA. You need to have your phone to receive it, surely?

The problem is in that assumption, that you need to have the phone to use the OTP. You don’t; it could have been intercepted in transit, or you might have entered it into a phishing site, or you set up your phone so that SMS messages are diverted to a mailbox and just protected by a password…

Effectively this form of authenticator is just a complicated form of “something you know”.

Factor strength.

We’ve all seen websites that tell you rules for setting up a password. This is based on the idea of making this factor “strong”. After all, a password that’s 50 characters long and written in emoji is stronger than “hello”… right? Unfortunately many of these rules forget that humans have to remember the password, and could be counter-productive.

However the concept of the strength of a factor is important. Multi-factor authentication requires strong factors; but strong is relative to the use case.

Let’s take PINs on an ATM card; that may only be 4 digits long, which sounds really weak! But in this use case the card may be rejected (or even retained by the machine) after 3 failed attempts. So our theoretically 10,000 entries has a 1 in 3,333 chance of being guessed. That seems small, but it’s actually pretty good. If we increased the minimum PIN length to 6 we may be 100 times stronger, but now its a lot harder for people to remember and so there’s a greater chance that it’ll be written down and kept in the wallet along with the card.

On the other hand, 4 digit PINs may not be suitable for a OTP sent as part of a 2SV process and we don’t need to remember this for a long time, so longer 8 digit PINs may be acceptable.

The use case is important in determining the strength of a factor.

Conclusion

A good MFA solution can strengthen the security of an environment; even a 2SV solution (e.g. Google Authenticator 2SV) is much better than just requiring a password. I use Google Authenticator on my own servers if coming from an untrusted source.

A large enterprise should require 2FA for staff entering their network from the internet (eg setting up VPNs, or remote desktop). For an individual you may wish to setup 2SV on your email account (especially since password reset emails are sent to them!). For your retro-computing web forum, though, you may not want to go to the effort.

I’d like to see FIDO U2F get more adoption; Firefox doesn’t seem to have it yet, but it looks like more vendors are planning it. It’s a cheap, simple solution to the “something you have” problem.

Encryption vs Hashing

Sun, 09 Apr 2017 17:51:36 -0400

Let’s say you run an ecommerce web site. As part of this people create accounts. Naturally those accounts have passwords. Do you encrypt those passwords before storing them in the database?

All you people who put your hands up and answered yes… put them down; you’re wrong. You should hash the data instead. This may sound like a “tomato/tomato” thing, but it’s not.

Encryption and hashing may both be cryptographic in nature, but there’s a subtle difference between them.

Encryption

Encryption is the act of taking a plain text message and encoding it. This process is reversible; someone with the right knowledge could take the cipher text (as the encoded data is known) and decrypt it back to the original plain text.

An early example that is sometimes taught to school kids is the “Caesar Cipher”. Old usenet nerds may know of a special case of this; “ROT 13”.

With the Caesar cipher a number is picked, and the characters moved forward that many. So, for example, if the number was “3” then we would move “A”->“D”, “B”->“E”, … “X”->“A”, “Y”->“B”, “Z”->“C”.

In this scenario the plain-text “HELLO THERE” would become “KHOOR WKHUH”.

To decrypt that string you would need to know the magic number (“3”), and then rotate backwards.

Now that’s a pretty simple cipher, and you wouldn’t use it in the real world!

Today you would use algorithms such as AES256 to encrypt the data, which is the AES algorithm with 256bit key sizes.

Use cases

Typically encryption is used when the original plain text needs to be protected and be recoverable. A password manager would use encryption to protect my passwords from being stolen. Only someone knowing the encryption key (me!) would be able to decode the values and get my password back.

Symmetrical vs asymmetrical.

Algorithms such as AES are considered symmetric because the same key is used to encrypt and decrypt the data. If you wanted to share data with someone else then they also need to know this key. Thus the key is a shared secret.

Asymmetrical encryption uses two different keys, which have a relationship to each other. A common one is the RSA algorithm. The public key is used to encrypt the data. It normally doesn’t matter that the public key is known (hence the name). The data can only be decrypted if the private key is known. This is the basis for SSL/TLS encryption used on the internet.

Symmetric and Asymmetric working together

Many asymmetric algorithms are slow and not always suitable for large chunks of data. RSA, for example, can’t encrypt data larger than the key size. In situations where we want asymmetric (e.g. the web) what we can do is pick a random key and use a faster symmetric solution for the data. We use the slower RSA system to encrypt that random key, and both the encrypted key and the encrypted data are sent to the other person. Using their private key they can decrypt the random value and then use that to decrypt the main data.

Salting

One problem with many cryptographic schemes is that the same plain-text will lead to the same cipher text. This means that an evil person looking at the messages might be able to see patterns; if the word “OK” is encrypted the same way each time then that evil person can make a guess.

To avoid this a salt value can be prepended to the plain-text before encrypting. This salt can be random; it really doesn’t matter what it is. Now the same plain text (“OK”) is made into a different message each time, and so the encrypted data is different each time.

Hashing

So if we need to use encryption to get the data back, why did I say encryption was wrong for storing customer passwords at the beginning of this post?

Hashing is a one way function. The idea is to come up with a piece of data that represents the original text. In this scenario the plain text is converted to a hash value.

A simple case, if we said that “A”=1, “B”=2, “C”=3… then we could define a simple hash of just adding up the values. Now the word “HELLO” would be hashed as 52 ( 8+5+12+12+15 ).

There is no way of getting back from “52” to the original plain text, which is why these are called “one way” or “trap door” functions.

Use cases

There are some common use cases for file integrity and checking. For example, if I have a file I could generate a hash value for it. Now at any point in the future I could re-compute the hash of file; if the hash value is the same then I can be confident the file has not changed. Similarly if I send you a copy of the file then you can compute the hash and if it matches my copy then you can be confident you’ve received the file.

The other main use case are for passwords. We don’t actually need to store the customer password in our database; we can store the hash value. Now when the customer wants to login, we can calculate the hash of their input; if it matches what we have saved we can be confident they entered the right password.

Collisions

Now in the use cases I said “confident”…

The earlier hash algorithm is bad and many strings would cause the same result (“OHELL” would also give 52). But if we pick a good hash then we can give a probability of how likely it is for someone to enter a different value that collides. For example, the md5 algorithm returns a 128bit number, which means that the chances of two pieces of plain text colliding are 2^-128. If you had 2⁶⁴ pieces of plain-text then you might expect to find a collision; a very small chance indeed.

Unfortunately, the md5 algorithm has a number of weaknesses, so it’s no longer considered a strong hash. Don’t use it.

With the right algorithm (eg SHA256) the risk of collision is considered so tiny that the chance of someone entering the wrong password on a login screen and getting the right hash is so vanishingly small that we can ignore it.

Salting

There’s another problem with hashing, though.

If I manage to get a copy of your hash file then I could try and guess what plain text was used to generate the hash. This is relatively simple for password files because people tend to pick easy to remember passwords. I could simply brute force attempt a lot of passwords and see what ones match. But there’s a simpler way; since the hash has to be consistent for a given plain text we could precompute all the common passwords and store the results. The result is called a rainbow table. Sure, they can be large but storage is cheap.

Now, with a rainbow table, converting from the hash back to a password is as simple a looking up the hash value.

To make this harder we can use a salt value. This is randomly chosen each time. For SHA256 this salt is 16 characters, resulting in the same plain text being hashed in one of 2⁹⁶ different ways. This can make rainbow tables impractical, and require the attacker to use brute force.

Now modern GPUs can make brute force attacks fast - over 4.7 billion md5s per second! Even SHA-1 is fast; it may take hours per password… but that’s not much money on an AWS-GPU cluster for a determined attacker!

Fortunately new algorithms (eg bcrypt and PBKDF2) have been designed to be slow to calculate. It doesn’t matter if it takes your login page an extra second to calculate the hash; it won’t do it very often! But slowing down brute force attacks can make them impractically expensive… today.

Update August 2020: PBKDF2 is no longer a recommend cipher. It’s weak-ish against GPU and ASIC attacks, which is the same sort of kit bitcoin miners use. Instead the recommendation is to use Argon2 or scrypt. This is a space that keeps changing as we learn more.

Summary

So if hash files and salts are not resilient to brute force attacks, why did I recommend hashing for customer password files? Simply because encryption requires the server to have a copy of the decryption key available (unless you have a HSM), so theft of an encrypted password file would also likely lead to theft of the decryption key, making it even easier for the attacker.

When you don’t need to recover the plain text, hashing is normally always a better solution. Just remember to pick a good algorithm that’s resistant to brute force attacks!

Role Based Access Control

Sun, 02 Apr 2017 15:14:14 -0400

Identity and Access Management (IAM) historically consists of the three A’s

Authentication
- What acccount is being accessed?
Authorization
- Is this account allowed access to this machine?
Access Control
- What resources are you allowed to use?

Companies spend a lot of time and effort on the Authentication side of the problem. Single signon solutions for web apps, Active Directory for servers (even Unix machines), OAuth for federated access to external resources, 2 Factor for privileged access… there’s a lot of solutions around and many companies know what they should be doing, here.

Well, small caveat… we’re pretty good at centrally managed infrastructure, but there’s a tendancy for application developers to try and introduce their own authentication store because it makes their life easier; keep an eye out for that! It will cause problems down the line; when Harry leaves the company have you found all the places his account may be stored? Do all these places follow your controls around password management? Do they follow properly audited access processes? And so on…

Why do we need RBAC

The problem starts to happen when we look at Authorization and Access Control. There’s a tendency to grant access on an individual basis, so you end up with “Tom can login to Servers 1,2,3,4; Dick can login to Servers 1,2,3; Harry can login to Servers 1,2,4”.

On the face of it, this looks OK… each person has the access they need to do their job, and no more.

But do they? Tom, Dick and Harry are all colleagues; they work together. Tom has called in sick and Dick is covering… except Dick can’t login to Server 4. Oops!

When Fred joins the team, what servers should he have access to? One “work-around” is to copy the access of someone else in the team (who?) but this can still break when the company gets Server 5; who gets access to that?

This problem gets massively worse as the number of servers and number of people grow.

RBAC

There were some hints in the previous section on how this can be solved; we can define roles. “This is the SA team”; “The SA team can login to Servers 1,2,3,4”; “Tom, Dick and Harry are SAs”.

We end up with a simple configuration:

Role Description
Role Members
Role Permissions

When Fred joins the team he gets added to the “SA team” membership and, magically, he has access to the servers he needs.

People can also be in more than one role. This tends to happen when people move roles or have multiple responsibilities; for example some may be a developer for application A and application B; he would be in both of those roles at the same time.

Servers have roles too!

The previous section mentioned “SA team can login to Servers 1,2,3,4”. Imagine we also had “DBA team can login to Servers 1,2,3,4”. When Server 5 comes along we need to work out all these rules and add the server to it.

Instead we can say “This is a Unix Server” “This is a Server for Application A” “This is an Oracle Server”

These are, effectively, roles for servers. So now when Server 5 arrives we can add it to the correct roles based on what it is going to be used for.

Results

Now we can have rules that say things like

          "Corporate Unix SAs" "can login to"     "All Unix machines"
          "Corporate Unix SAs" "can NOT login to" "Application A Machines"
           "Application A SAs" "can login to"     "Application A Machines"
    "Application A Developers" "can login to"     "App A Dev machines"
                 "Oracle DBAs" "can login to"     "Oracle servers"

It’s a pretty simple list. We can extend this to access controls on the machine as well; eg

"Application A SAs" "can run as root" "/bin/ls" "on Application A Machines"

Implementation

This is where things get a little harder. From a technology perspective, a role maps quite nicely into a group. In the old Unix days we had Netgroups to control this sort of thing. With Active Directory, security groups. Or you may want to bring in third party tooling to help manage this.

The problem is in defining the roles. And you will get this wrong. Everyone does!

There is a temptation to minimise disruption when going from a disorganised authorisations model to an RBAC one; we can attempt to discover roles, based on actual activity, perhaps role up via the HR database to a consistent manager. Tools exist to try and assist in this data analysis and third party organisations are willing to provide professional services to assist in this (for a fee, of course!).

Alternatively you may want to “start fresh” and build a parallel structure to your existing AAA solution, define the roles and then migrate servers into it (e.g. application by application). This can take more time, but may result in a cleaner structure; we haven’t discovered roles; we’ve defined them.

Either way there’s a transition period and a testing period. You won’t do this in one weekend!

Other benefits of RBAC; auditing!

If you’re in a controlled environment with external audit constraints then a typical question asked is “who can login to your application servers, and why?”

RBAC makes this easier to understand…

My server is a “Unix machine” - Unix SAs can login to them.
My server is a “Database server” - DBAs can login to them.
My server runs Application A in production - AppA support can login to them.

Now we can look into these groups and get the list of people.

If the auditor asks “Why can Harry login this server” we can say “Harry is a Unix SA; Unix SAs can login to my servers”.

It becomes very easy to explain to auditors the who and why. And the results are consistent; there’s no “Why can Tom login to server 4, but Dick can’t?”

Beyond RBAC - ABAC

RBAC is a simple model, but it requires role maintenance. When someone joins the company we need to have him added to roles; when someone changes job we need to verify their new roles; we typically need regular recertification that roles are still accurate.

Attribute Based Access Control (ABAC), on the other hand, grants access based on things we know about.

Tom is based in New York
Tom has job code 12345
Tom is a US Citizen
Tom has passed enhanced vetting
Tom reports to John
...

With ABAC we can write rules based on those things:

People who are based in the USA *AND*
       who are US Citizens or Green Card holders *AND*
       passed enhanced vetting *AND*
       have job code 12345
Can access Unix servers THAT
           are running government applications *AND*
           are running Oracle

That’s one complicated rule! But now it’s pretty much self-supporting; all the information comes from upstream systems (e.g. the HR database, the CMDB). When Tom changes role, his job code changes and so he automatically loses access. Of course we rely on good data from those upstream systems…

Now very few systems currently support ABAC; they’re group based and thus RBAC. It’s not uncommon to have batch jobs that run regularly and try to build RBAC groups out of ABAC rules. This isn’t as clean as a full ABAC solution, but it allows you to write rules in an ABAC manner while still working with RBAC technologies.

Summary

It’s surprising, to me, how many organisations are still at the “unique access rights” stage. It just makes life hard, and there’s no consistency of access and control.

RBAC requires a management layer to process it properly, but when this has been achieved then it becomes a lot easier to manage and control access in an environment.

Of course, if you only have one security boundary (everyone can access everything) then such a process is just unnecessary overhead!

Lessons from a pentest run

Sat, 25 Mar 2017 15:06:47 -0400

Over on Twitter, @TinkerSec live tweeted a pentest and created a moment thread of it.

It’s fascinating reading, and well worth reading. Even non-technical people should be able to get something out of this. I like that it’s a form of insider attack (industrial espionage by a newly hired employee? disgruntled employee? vendor allowed unaccompanied access?) rather than an external attack.

One of the things that typically comes out of an event like this is a series of action items. I could imagine the following bullet points on some powerpoint somewhere

User education; remind them to never give out passwords or 2FA numbers to anyone. Possible training course?
USB lockdown; disable storage devices totally except for people on an exception list
Time based One Time Passwords for IT infrastructure accounts
Review all devices checking for disk encryption
Lock support areas

You get the idea. A nice solid list of action items that all sound reasonable.

The problem with such lists is that they end up focusing on one attack path and closing that down. We can see how Tinker changed tactics during his test; one path is blocked, so try another. If we shut down the path he eventually used, would another one be found (block USB storage; use a rubber ducky instead?).

Aside: many Virtual Desktop (VDI) solutions provide some level of control here; e.g. if you log out of your VDI session then the VM is destroyed and next login is on a newly created VM, which makes unquoted path attacks harder (your code doesn’t persist between logins). Similarly thin desktops typically don’t allow random USB devices access (which I find annoying; it means I can’t use Ubikey U2F devices for 2FA).

This is not to say that prevention is a bad thing; it can raise the bar and require smarter attackers. It can prevent accidental data breaches (e.g. someone copies important data to a USB stick, loses it on the train) but you can always go too far down the rabbit hole and end up doing the equivalent of destroying a $200 drone with a $3mm missile. Your time, effort and money may be better spent elsewhere.

One such area is in detection. And, indeed, this is ultimately what caught this attack; anomalous behaviour on a desktop (base64 encoded powershell, of all things) led to the compromised machine being isolated and security called.

Intrusion Detection Systems (IDS) are the partners of IPS (Prevention). They can highlight where IPS has been circumvented or avoided. It can take place in many places (network demarcation points such as firewalls, routers; endpoints such as desktops and servers; central infrastructure tools such as scanners and integrity monitors). It should even exist at the application layer (anomalous activity).

IDS work better the more data they have so it is common to start feeding logs into a central Security Operations Center (SOC). A common form is just to log everything to Splunk, and then let the SOC deal with it.

But an IDS is pointless if there’s no knowledge behind the data. A tonne of data streams results in just noise. We need to know what is normal behaviour; what is expected; what thresholds are normal (we expect 50 failed logins a minute from a user population of 10,000; if we see 500 then there’s odd stuff happening). This requires data modelling. Tools exist to try and “self-learn” baseline activity, but these may sometimes get confused and learn the wrong thing. So teams onboarding to the SOC must work to ensure their data is understood as information. The SOC needs to be aware of all their datastreams and potentially how the interact in order to gain knowledge.

And then, finally, once suspicious activity has occured we need the ability to take action. Isolating a rogue device is good, but may not be viable everywhere (“our IDS detected suspicious activity from server X, so we shut down the network port… oh, did we just break our primary trading platform for 2 hours at the cost of $$$$? Let me pack up my desk…“) so the response needs to be tuned to the type of activity and risk to the company.

As we saw with IPS, there will always be gaps in coverage. A common refrain is “red teams have it easy; we have to close every security gap but they just need to find one.”

Tinker turns this on its head; “Blue Team has it easy. All they need to do is find ONE IoC. Red Team? We have to be perfect in every attack to not get caught.”

As with IPS, IDS coverage won’t perfect. Things will slip through.

Summary

Using tools and processes are an essential part of preventing an attacker from succeeding in the first place. However many responses to a break are focused on the immediate issue. To use an ITIL term; they are incident response focused, rather then problem management focused. Of course no one likes having a known vulnerability, so we will try and close that gap, but sometimes we need to look at the bigger picture.

First rule of security: your system will be broken into. So let’s also focus on setting up tripwires and alarm bells. A bank vault is hard to break into (IPS), but we also have alarms that ring (IDS). Detecting a breach is just as important as preventing them in the first place.

Phishing and Certificate Transparency

Sun, 19 Mar 2017 13:23:00 -0400

Many people are at a large risk of a phishing attack. In this scenario the person may receive an email that looks like it came from a legitimate source (e.g. their bank) and encourages them to click a link that presents them with their bank login page. The user then attempts to login…

Except that site isn’t their banking site. It’s a mockup that looks like the real one. And they’ve now told their banking password to the attacker.

A variant on this, called Spear Phishing, is specifically targetted at a company; e.g. the website might look like an intranet login page, and now the attacker has gained credentials to the company network; a situation made worse by SSO solutions; that password may be the same password used to login to your Windows desktop!

Related to this (and playing into this) is Typo Squatting; the bad website might be called paypale.com, for example. Or paipal.com or paypa1.com or… the ways the URL can be made to look similar to the original are unlimited. They need not even be that smart; paypal_com.example.com?

SSL as a mitigation… or not!

In the early days of SSL it was very hard to get a certificate; you had to prove to the Certificate Authority (CA) who you were, and there were checks in place to prevent bad certificates from being issued. We also wanted to encourage people to only enter sensitive data, such as credit card numbers, to an SSL encrypted site. So we taught people to look for the “green padlock”. If there was no padlock then it is not safe.

And this kinda worked, for a while. The difficulty in getting a cert meant that a green padlock was a level of trust. And so this advice mutated and started to be understood as “green padlock means it is safe”. Which is not quite the same thing.

With automated free certificates, such as those from LetsEncrypt I could request and get a certificate for www.paypal.com.sweharris.org. In this scenario a phisher could send potential victimes to a site that has the green padlock.

People learned the wrong lesson from the green padlock.

Now there are some things we can do to help teach people (many companies use services such as PhishMe to run training and test scenarios. The use of Extended Validation Certificates can help highlight good sites. And, of course, teach people to not click links and carefully inspect URLs before entering data.

But even the best of people make mistakes. I may have been phished once, but the second after I realised this I changed the password on the account. If it can happen to me then it can happen to anyone.

Ideally, of course, CAs wouldn’t release certs for well known domains (what legitimate reason do I have for paypal.com.sweharris.org?) but given the sheer number of sites in question (banks, other financial institutions, government sites, commerce sites…) and the typosquatting and obfuscation problem, this is not really practical.

The consequence of making it easy for the ‘good guys’ to get SSL certs means that it’s also easy for the bad guys to get them as well.

So what can companies do to protect their customers?

Certificate Transparency

Certificate Transparency (CT) came about as a result of a number of errors made by CAs, such as mistakenly providing a cert for a domain they shouldn’t have done, or a hacker compromising a CA partner to gain certs for google.com and similar.

With CT every certificate issued is logged to a number of log services by the CA, and those logs are open to the public.

One easy to view site is crt.sh. This is an easy to use service. You can enter the domain name you care about and get a list of all the certs issued.

For example, the certificates for this site:

Hmm, interesting… why did I request a new cert on 2017/1/29, just 5 days after the previous one? Digging deeper I can see that’s because I added this site as a SAN entry to my default (non-SNI aware) cert.

Automation

Of course these sorts of logs are fine for “eyes on glass” monitoring but larger companies don’t want manual processing; automation is the key. Fortunately there are APIs available as well. Symantec run a log that produces nice JSON output that is easy to call.

eg the call for this site would be:

https://cryptoreport.websecurity.symantec.com/chainTester/webservice/ctsearch/search?keyword=sweharris.org

A typical entry might look like

      {
        "documentId": "q1SpPQpqDh6nQoKQW5sl5CjPNbwBlMY6F7LE1K7vakQ=",
        "serialNumber": "03483b8d546d95fe3c1726b5f68eb5b9533a",
        "commonName": "www.sweharris.org",
        "notAfter": "2017-Feb-10",
        "notBefore": "2016-Nov-12",
        "organization": "",
        "issuerCommonName": "Let's Encrypt Authority X3",
        "issuerOrganization": "Let's Encrypt",
        "subjectAlternativeNames": [
          "sweharris.org",
          "www.sweharris.org"
        ],
        "issuer": "Let's Encrypt Authority X3"
      },

It now becomes possible for a company to automate checking of CT logs and compare them to their request system. Any log entry that shows up that doesn’t match a request can be flagged to the Security Operations Center (SOC) for followup and investigation with the CA.

Companies can also be proactive. They can look for re-use of their name elsewhere (so Paypal could search for any cert that has paypal in it). They can also look for common typosquatted variations (although, as previously noted, this isn’t a complete answer). This can help mitigate some of the phishing attacks attacks.

Summary

CT logs are one way that a company can protect their customers from a number of phishing attacks. It’s not perfect, but it has benefits.

Enterprise key management solutions may offer the ability to search CT logs as part of their solution; if you’re looking at such a solution then ask the vendor what they do about CT logs, and how integrated it is in their solution. They may even run their own CT log server!

Other tools, such as CAA DNS records (a method of telling CAs which ones are authorised to create keys) and HPKP to pin known good keys into the browser can also help out with rogue certificate issuance (Comodo would refuse to issue a cert for your domain if your CAA records says that only letsencrypt is allowed!) but CT logs are a way of verifying what has been issued.

Offsite Backups in the cloud

Sun, 12 Mar 2017 15:26:18 -0400

Part of any good backup strategy is to ensure a copy of your backup is stored in a secondary location, so that if there is a major outage (datacenter failure, office burns down, whatever) there is a copy of your data stored elsewhere. After all, what use is a backup if it gets destroyed at the same time as the original?

A large enterprise may do cross-datacenter backups, or stream them to a “bunker”; smaller business may physically transfer media to a storage location (in my first job mumble years ago, the finance director would take the weekly full-backup tapes to her house so we had at most 1 week of data loss).

With the internet now being so fast (my home connection is faster than my LAN was 10 years ago!) offsite storage in the cloud is now a feasible option, and a number of businesses have started up around this concept allowing anyone, even home users, to have an offsite backup (e.g. Crashplan, Carbonite, Backblaze). Many of these solutions require an agent to be installed on your machine and it will “live” detect changes and upload them to the cloud storage, which is nice.

These services are priced in a number of ways (e.g. fixed price, or per Terabyte). They may even allow you to “bring your own storage” (e.g. backup to an Amazon S3 pool you control). You should also be aware of network transfer charges.

The DIY solution

I’m an old-school Unix guy and do old-school backup solutions. So each server has a nightly job that does a dump (level 0 on Sunday, level 1 on Monday, etc). For my remote servers I rsync the data back to my home machine. I have an LVM on here that stores all the backups for the past 2 weeks. This works fine for data restores, but would be vulnerable to my house burning down (for example); backups of my home machine and the original data are in the same location.

Until recently I had a physical server colocated in Canada and would copy my files to that machine. However I cancelled this service (it was just too expensive and I didn’t need a full server; my existing virtual machines did the job I needed) and started looking for an alternative way for handling offsite files.

I had spotted that Amazon now have a Cloud Drive. What makes this service attractive is that the $60/year option is unlimited. I’ve read that people have stored over 100Tbytes of data here. Options such as Google Drive charge $20/100Gb/Yr (or $100/Tb/yr) so if you store over 300Gb of data then Amazon is cheaper. Similarly Dropbox is $100/Tb/yr. This makes Amazon nice from a pricing perspective, but it’s not necessarily so well supported as other options (eg Backblaze have dedicated customer support for their backup/restore solutions and can even ship you a drive with your data on… for a fee, of course!).

Now Amazon Cloud Drive (ACD) has agents for Windows and Mac but, of course, not for Linux or Unix in general. Fortunately people have coded to the ACD API, and provided solutions such as acd_cli or rclone.

rclone looked attactive to me; it can act like rsync but for Amazon (and other) cloud drives. In addition it can also encrypt the data sent to the cloud which, to me, is pretty important; I don’t want to become a statistic like Capgemini and expose my backups to the world!

rclone configuration

Once I’d signed up for ACD (oh, hey, 3 month free trial so if this doesn’t work out then I can cancel it!) it was pretty simple to configure rclone to talk to Amazon. The rclone config comamnd steps you through the process (even on headless servers) and builds the config file:

[Amazon]
type = amazon cloud drive
client_id = 
client_secret = 
token = {"access_token":"***","token_type":"bearer","refresh_token":"***", "expiry":"***"}

And that’s all there is to it. So now we can work directly from the comamnd line:

$ rclone mkdir Amazon:foo
$ rclone copy ~/.profile Amazon:foo
2017/03/12 16:08:25 amazon drive root 'foo': Waiting for checks to finish
2017/03/12 16:08:25 amazon drive root 'foo': Waiting for transfers to finish
2017/03/12 16:08:26 
Transferred:   1.714 kBytes (1.715 kBytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:       900ms
$ rclone ls Amazon:foo
     1755 .profile
$ rclone delete Amazon:foo/.profile
2017/03/12 16:08:41 Waiting for deletions to finish
$ rclone rmdir Amazon:foo

This is pretty easy! And, of course, the files are visible from a web browser.

Encryption

So far this data isn’t encrypted, so now we need to create an encrypted area. The nice part of this is that you pick a subdirectory and specify that for encryption. Again rclone config can build this for you; the resulting config section looks like this:

[backups]
type = crypt
remote = Amazon:Backups
filename_encryption = off
password = **THATWOULDBETELLING*
password2 = **THATWOULDBETELLING*

The remote line tells rclone to use the Backups directory on the previously created Amazon setup.

An interesting line is filename_encryption. This can be off or standard. In off mode the filename appears unchanged (well, .bin gets added to the end). In standard mode the filename also gets encrypted. For my backups I didn’t care if people saw linode/20170312.root.0.gz.bin as the filename, and it makes it easy to see what is there via the web.

I can now do rclone copy /BACKUPS backups: and the program will recursively copy all my data up.

My home connection is a FIOS 75Mbit/s link. Uploading averaged around 60Mbit/s, so it took a while to upload 320Gb of data, but I left it overnight and it worked.

So now let’s see how this looks:

$ rclone ls Amazon:Backups/linode/linode | sort -k+2 | head -3
     6358 20170223._datadisk.4.gz.bin
 80846055 20170223._news.4.gz.bin
     4214 20170223.log.bin

$ rclone ls backups:linode/linode | sort -k+2 | head -3       
     6310 20170223._datadisk.4.gz
 80826279 20170223._news.4.gz
     4166 20170223.log

We can see the filesizes are larger when looking at the raw data; that’s part of the encryption overhead.

Verify

Of course a backup isn’t any good if you can’t restore from it, so I also tried downloading all the data as well. I created another temporary LVM and then restored all the data backup

$ rclone copy backups: .

[...]

Transferred:   341.269 GBytes (9.703 MBytes/s)
Errors:                 0
Checks:                 0
Transferred:          706
Elapsed time:  10h0m17.2s

That looks good!

$ diff -r . /BACKUPS
Only in /BACKUP/: lost+found
Only in /BACKUP/penfold: 20170226._FastData.0.gz

That’s interesting… we learn two things:

Empty directories are not copied over
Amazon has a maximum filesystem limit; currently this is 50Gb. Any file larger than this won’t be copied.

The second entry has meant I’ve modified my backup software slightly; it now splits the data files every 10Gb to ensure they’re small enough.

But other than that, the data I got back was identical to the data I sent up.

Worst case scenario

When I wrote, earlier, about backups I mentioned the need to be aware of what software you need to perform a recover.

In this case I need two files; the rclone binary and the $HOME/.rclone.conf configuration. Now the binary I can store in the Amazon drive as well; if I need it then I can download it via a web browser.

More critically is the configuration file; this contains the encryption strings for my data, and so I don’t want this easily visible. So this file I stored in my password manager (lastpass).

So now all I need to bootstrap my recover process is a web browser and knowledge of the lastpass password.

Simplified restore

In the above tests I copied all the files back before comparing. However, rclone also has experimental FUSE support for mounting. It works best for reading. So now I could do:

$ sudo /bin/fusermount -u /mnt/Amazon
[1] +  Done                    sudo rclone mount backups: /mnt/Amazon &
$ 
$ sudo rclone mount backups: /mnt/Amazon &
[1]     29086
$ sudo diff /BACKUP/linode/linode/20170312.log /mnt/Amazon/linode/linode
$ sudo /bin/fusermount -u /mnt/Amazon                                         
[1] +  Done                    sudo rclone mount backups: /mnt/Amazon &
$

Direct access to the files without a lot of temporary storage needed! Which might be important after a disaster, if I’m short of disk space :-)

Behaviour change

Media storage

Unlimited storage has a few other benefits as well. I have ripped all my CDs to FLAC format. That’s another 300Gb or so. If my system died I’d be annoyed to have to re-rip all the CDs (which may not be available after a fire). But with unlimited storage…

In this case I don’t want Amazon to see the filenames, just in case they have some automated scanning tool looking for potentially pirated content (I do own these CDs, but would an automation tool know that?).

In this case the config entry looks like

[flac]
type = crypt
remote = Amazon:My_CDs
filename_encryption = standard
password = *IWONTTELL*
password2 = *IWONTTELL*

Now I can rclone sync my FLAC files up. Encrypted filenames look a little odd:

$ rclone ls Amazon:My_CDs | head -3
 30038497 c87a5pke85pd9ev7gp52aftecj8eokp19p2gn900pkkdb9ngr4n0/042hle8jn609itgohdh0q4gt6q7b3smbndn1cfjkklnsm2qm5l06u7au2ufme7jgne1oc8c7b68j4oarc9ebgks6faufrdeel3qv5j0
 32853891 c87a5pke85pd9ev7gp52aftecj8eokp19p2gn900pkkdb9ngr4n0/p705nk3k7j9ns64ki3bf7go7c8jq9q439j2avlcu6ipbpi4qvjmg
 23424207 c87a5pke85pd9ev7gp52aftecj8eokp19p2gn900pkkdb9ngr4n0/bf8e4mj9d7184f3021mdhbgua876d40stfa18ao5kfjl2sinaie805qc12r86f1jc32flct9ova8i

Again I verified the data I uploaded looked correct.

Amazon now claims I’m using 650Gb of storage.

I’m now thinking whether I should also upload my ripped DVDs as well. I don’t see why not…

Unlimited backups

Previously my backups retained only 14 days of history. The overnight process would delete files over that age, to keep disk usage in check.

But with unlimited storage I could start keeping offsite backups for months, if not years.

This also helps with any possible ransomware issue; if something did encrypt my local files and corrupted the backups but kept the sync process unbroken then the last 14 days of offsite backups might get broken, but older files won’t show locally and so won’t be touched. Annoying but not critical.

Automation

With the backups and music now available, I can automate the offsite copying as part of the overnight job:

#!/bin/ksh -p

rc()
{
  echo
  cd /$1 || exit
  /home/sweh/bin/rclone --stats=0 --max-size=40000M --config /home/sweh/.rclone.conf $2 . $3:
}

rc /BACKUP copy backups
rc /Media/audio sync flac

Note the the backup uses copy (so new files get copied up, old files get left) but the music uses sync; if I sell or otherwise get rid of a CD then I delete the flac files and the offsite copy will also get deleted.

Today was Sunday, so a level 0 backup. The output from the job reports that it took just over 4 hours to upload all the backups:

2017/03/12 12:46:51 Encrypted amazon drive root 'Backups': Waiting for checks to
 finish
2017/03/12 12:46:51 Encrypted amazon drive root 'Backups': Waiting for transfers
 to finish
2017/03/12 13:56:15
Transferred:   117.701 GBytes (7.795 MBytes/s)
Errors:                 0
Checks:               650                                   
Transferred:           48    
Elapsed time:  4h17m41.2s

The daily incremental typically takes under 7 minutes.

Limitations

rclone and ACD have some limitations. These are what I’ve hit, so far:

No files larger than 50Gb
Empty directories are not copied
Symbolic links aren’t copied (the latest DEV version has an option to follow links, but can’t copy the link itself)
Large filenames (eg over 160chars) break if using filename encryption

Conclusion

Cloud storage for offsite backups are now a viable technology… if you have a fast enough internet connection. Beware of hidden costs in your tool of choice and make sure you can recover from a complete failure of your hardware.

In my case rclone and ACD make a good scriptable solution.

There is now no reason for home users to not have offsite solutions, just like large enterprises!

Abusing LD_PRELOAD for fun and profit

Sun, 05 Mar 2017 18:10:03 -0500

The ELF format is pretty common across various unix versions, having superseded previous binary formats such as a.out and COFF. Pretty much, today, if you see a unix binary then it’s probably ELF format.

One of features of the ELF format is that the run time linker can be smart about how it resolves dependencies, and this smartness can be tuned. A typical tuning many people know is the LD_LIBRARY_PATH variable, which can be used to add new directories to be searched for the needed libraries.

Another one, the focus of this entry, is LD_PRELOAD. This can be used to force a library to be loaded and this can be used to alter the way the standard libc calls are processed.

This entry may get a little more technical than normal for this blog; I just felt the need to post some code! I’ll present three unusual use cases:

The snowflake

In small companies or self-managed departments we typically have usernames that are easy to use. So my login may be sweh or sweharris or stephen or similar. This is nice for humans, and makes it easy; an ls or a ps will show the username and the human looking at it will know what person is really being referenced. Unfortunately this isn’t scalable (what if you have two people named “Stephen”). Larger companies tend to give people generated names. The login name may be random or refer to the employee ID. Now xyz123ab or x123456 doesn’t really help.

I had to deal with a manager who ran servers that were being migrated from “self-managed” into the corporate management tool. Of course this meant the “fred” account now had to become “z987654” and he was angry. He demanded to be a unique and special snowflake and keep human friendly names. We weren’t going to do this, but it gave me an idea for a joke. I could create a snowflake.so library that could be pre-loaded. It would override the normal getpwuid() call and rewrite the results.

Ultimately, after the Managing Director told him to shut up, he gave up.

Now a full version of the code had configuration files, but here is a simple variant; it will replace the current user’s username with a variable:

/* This routine intercepts getwpuid()
 * It will call the original, and if the called
 * uid == getuid() and $OVERRIDE_USER is set then
 * replaces the username with $OVERRIDE_USER
 *
 * gcc -o getpwuid_modify.so -fPIC -shared getpwuid_modify.c -ldl
 * export LD_PRELOAD=$PWD/getpwuid_modify.so
 * export OVERRIDE_USER="dummy_username"
 * run the app
 * unset LD_PRELOAD
 */

#define _GNU_SOURCE
#include <sys/types.h>
#include <pwd.h>
#include <stdlib.h>
#include <dlfcn.h>

struct passwd *getpwuid(uid_t uid)
{
  static void * (*func)();
  static uid_t my_uid;
  static char *override;
  struct passwd *p;

  if(!func)
  {
    func = (void *(*)()) dlsym(RTLD_NEXT, "getpwuid");
    my_uid = getuid();
    override = getenv("OVERRIDE_USER");
  }

  p=func(uid);
  if (p && uid == my_uid && override)
    p->pw_name=override;

  return(p);
}

What this code does is intercept the getpwuid function. It then looks for the original function and calls that, then manipulates the result.

To the calling program we just get a different string:

$ gcc -o getpwuid_modify.so -fPIC -shared getpwuid_modify.c -ldl
$ export LD_PRELOAD=$PWD/getpwuid_modify.so
$ whoami
sweh
$ OVERRIDE_USER=root whoami
root

Let’s hope there’s no programs that test the result of whoami to determine what functions are allowed!

Shifting time

Recently, on twitter, a friend asked if there was a good way of setting his computer clock 5 minutes fast. I guess he’s one of these people who needs his alarm to start 5 minutes early. Now you don’t want to play with the system clock because NTP would fight you. The normal way would be to create your own unique tzdata entry (easily based off your local timezone). This will work for most people because their TZ rules don’t change often.

Or you can cheat and override the library calls :-)

/* This routine intercepts time() and clock_gettime() adds 300 seconds.
 *
 * gcc -o time_modify.so -fPIC -shared time_modify.c -ldl
 * export LD_PRELOAD=$PWD/time_modify.so
 * run the app
 * unset LD_PRELOAD
 */

#define _GNU_SOURCE
#include <sys/types.h>
#include <time.h>
#include <stdlib.h>
#include <dlfcn.h>
#include <stdio.h>

time_t time(time_t *tloc)
{
  static time_t (*func)();
  time_t t;

  if(!func)
  {
    func = (time_t (*)()) dlsym(RTLD_NEXT, "time");
  }

  t=func(tloc) + 300;
  if (tloc) *tloc=t;

  return(t);
}

int clock_gettime(clockid_t clk_id, struct timespec *tp)
{
  static int i;
  static int (*func)();

  if(!func)
  {
    func = (int (*)()) dlsym(RTLD_NEXT, "clock_gettime");
  }

  i=func(clk_id,tp);
  tp->tv_sec += 300;

  return(i);
}

We can see the code logic is very very similar and that multiple functions can be overridden.

$ date ; LD_PRELOAD=$PWD/time_modify.so date
Sun Mar  5 18:48:05 EST 2017
Sun Mar  5 18:53:05 EST 2017

I think he eventually went with timezone creation; it’s pretty easy for America/New_York :-)

Overcoming license restrictions

This one was big in the 90s. Sun SPARC servers had an NVRAM chip, and each host had a unique hostid value. Some licensed software had a license file that was based on this hostid. If you tried to run it on another machine then it’d fail to start.

A number of solutions were available for this, including reprogramming the NVRAM, but a common one was to use LD_PRELOAD.

In this case the code is a lot simpler; we don’t need to call out to the original routine or do anything conditional; just return the number we want. (the compile sequence is different here; it’s for Solaris)

long gethostid(void) {
 return 0x12345678;
}

$ gcc -fPIC -c newhostid.c
$ gcc -shared -o newhostid.so newhostid.o
$ export LD_PRELOAD=/tmp/newhostid.so
$ /usr/bin/hostid
12345678

Of course this was never used to allow license keys to be reused on multiple servers at the same time. Oh no, that would be wrong…

Summary

LD_PRELOAD isn’t really something I’d recommend for production quality code. I can see it being useful during development (“I want to test this new function without rebuilding a whole library”) and, as can be seen, it can be used to circumvent some forms of protection. But I wouldn’t consider it supportable!

And, no, you can’t use it against setuid programs; the linker ignores these variables in this case. You can’t make su give you a root shell by overriding library calls :-)

Can you control the entry points to your network?

Mon, 20 Feb 2017 19:12:53 -0500

One of the major threats that companies are concerned about is “insider threat”. According to some Data Breach Incident Response (DBIR) analyses, insider threat may be the 2nd or 3rd major reason for data loss. It’s interesting to note that the insider threat is way down in the actual number of incidents, but they count for a larger number of successful data loss incidents because the insider knows where the data is, may have legitimate access to the data, and may know the controls that need to be bypassed to exfiltrate it. Snowden may be the most well known of these types of incidents, especially in the US.

Related to this is the ex-Insider. Someone who used to work for you and no longer does. They also know where the data is, what controls you have, and may know passwords to access the data. When someone leaves, can you be sure that their access to data has been revoked?

A recent story can highlight what happens if a privileged insider is let go, but his access not fully revoked.

So how can we defend against this sort of malicious ex-Insider?

Network access

If you can’t get access to the network in the first place then it’s a lot harder to attack servers and services!

This person gained access via a VPN. It’s not stated, but looking at the indictment it seems as if this person had a company Cisco Linksys router; likely this was preconfigured to connect to the corporate VPN automatically, and this was disabled

Lesson: Ensure all remote access solutions for an ex-Insider are terminated immediately. If this is a user based VPN where the user needs to enter their username and credentials then ensure those credentials are terminated; if this is a device based VPN then ensure the device’s access is revoked.

Now a truly malicious person could have planned ahead and created their own backdoor network; eg a process on a server that makes outgoing tunnel requests via the web proxy, or a piece of software installed on a secretary’s desktop, or even a small device that plugs into the network (these can be so small now that they can plug in to a desktop port, and the real desktop PC plugged into that, and if you’re not looking carefully then you’ll never spot it). There are lots of ways that remote access malware can work; your malicious ex-Insider could use any of these.

So although we can control the front door, we can’t always determine if there’s a rogue backdoor. So we need further protection

Connecting to a server

Personal accounts

Typically you need to provide a credential to login to a server! If “Jim” wants to login, he’ll normally login with the “jim” account. So we need to make sure that Jim’s account is deleted. Many regulatory authorities use the word “immediately”, but in practice this may be triggered by the HR system and processed in batch overnight. However you need to have a manual process to perform emergency removals just in case that batch process is too slow. This, of course, means you need a good Identity and Access Management (I&AM) program. Whether you use Active Directory (and hook your Unix machines into that) or LDAP or even distributed password files, doesn’t matter; you need to manage them

Lesson: Delete ex-Insider personal accounts ASAP

Shared accounts

Even if you delete Jim’s account, does he know the passwords for any other accounts? Does he know the root account password? You better change that. What about the oracle account? sybase? The account your application runs under? Your router admin passwords?

These non-human “functional” accounts are a common weak point. They’re all privileged accounts, but the password is commonly shared amongst team members. If you have a good I&AM program then you may be able to change these passwords… but are you sure you know where they all live?

Ideally these accounts should have passwords stored in a password vault, requiring a “break glass” process if any human ever needs to know them; this process will change the password within a few hours of a human being told what it is.

Alternatively a “privileged access” gateway may be used; this can use the vault to automatically log a user in (using the user’s own 2FA credentials) and audit their activity. Now no human ever needs to know the password.

Lesson: Stop sharing passwords for functional accounts. Use a good privileged access management program to ensure that the ex-Insider’s knowledge expires quickly.

Alternate credentials

Not all login access is done via passwords. A common alternative is to use SSH public key authentication. This brings in a new set of challenges. Our malicious ex-Insider may have planned ahead and added his keys to another person’s account. Now, when Jim is terminated, he can use these keys to login as Harry. As far as the system is concerned, this is Harry. Deleting Jim’s account doesn’t help, now!

Lesson: Ensure any alternate authentication tokens are properly managed. If you’re using SSH public keys then you may want to look into a key manager solution to ensure that “rogue” keys haven’t been added.

Local “rogue” accounts

Related to rogue credentials, we may also have rogue accounts! It’s not too unusual for a lazy SA to create a second privileged account (“jimroot”) which he can access, bypassing the privileged access management system. They justify this to themselves as an efficiency gain (it is!) but it shouldn’t be allowed. However our malicious ex-Insider may also have planned ahead and created some of these accounts.

Lesson: A full I&AM system must also be able to track and audit any local accounts created outside of the central tool. They should be handled appropriately (deleted? disabled? brought into central control?) and shouldn’t be allowed to be kept uncontested.

Control systems

Typically environments also have non-PC/non-server environments. You may have IP telephony; your door access systems may be network connected; your building temperature control systems may be networked; your industrial control systems may be networked (see Stuxnet). Many of these systems can’t be controlled the same way as your servers and desktops; they’re not smart enough to hook into centralised systems or may have low-entry PINs that are shared. Such systems should be firewalled off from the main network and protected by strong “jumphost” servers, which may require 2FA to ensure the authenticating user has permission to access the device (and also log their activity)

Lesson: Weaker devices should be network segregated and protected.

Summary

I’ve only covered a small subset things that can be, and should be, controlled. By focusing on this area (I&AM, Privileged access, jumphosts) we can mitigate a lot of the risk of an ex-Insider. At the same time we’re also mitigating the risk of a current insider attack because we’re handling privileged access at the same time!

This isn’t to say we shouldn’t also have additional controls in place around server changes (stop a malicious insider from planting bad code), network access controls, external gateway monitoring (web proxies; DNS servers; etc). There’s a lot that can be done there as well.

The number one control is to stop an attacker from getting a foothold on your network at all. However if that does happen (and it will happen) then we can make sure that we have secondary controls, such as proper I&AM systems, to limit the attack - especially from an ex-Insider.

Managing the cloud management layer

Mon, 13 Feb 2017 14:12:32 -0500

A typical cloud engagement has a dual responsibility model. There’s stuff that can be considered “below the line” and is the responsibility of the cloud service provider (CSP) and there’s stuff above the line, which is the responsibility of the customer.

Amazon have a good example for their IaaS:

Where the line lives will depend on the type of engagement; the higher up the abstraction tree (IaaS->PaaS->SaaS) the more the CSP has responsibility.

But there’s a part of this that is frequently missed, and that’s the line itself. This line is typically where the CSP exposes controls to the customer (“self service” is part of the cloud model). For Amazon this may be the console where you can spin up a new VPC, new VMs, change firewall rules, modify VPC-VPC communication and so on.

This is clearly critical for a secure cloud offering. What use is there in defining a firewall if the console can be used to create a new VM with a connection to the internet and bypass this firewall?

We need to manage the management layer. Otherwise this layer can be used to bypass controls and create a shadow IT environment.

What this means will depend on your organisations stance and controls; is an audit and reporting control (detective) sufficient or do you want pro-active automated control over the management layer? Can your CSP provide the automation you need?

For an enterprise we may need to consider some questions:

Who has access to this management layer (privileged access)
Who has access to resources (end users)
Who logged in?
Who failed to login?
What activities were performed
What is the current configuration state

You’ll note these are the same sort of questions that an application control process would also ask; that’s not a surprise! We’re managing the cloud management layer the same as an app :-)

Some of these may be managed by Federation (can your existing authentication solution federate out of your datacenter?), or via synchronisation (hourly job that compares what users have permission to who should have permission), or (worst case) by a human eyeballing a screen.

Can your log infrastructure be reachable from the CSP (maybe a gateway service)? Can events be streamed into it, or pulled programmatically?

The breadth of the controls made available by the cloud service provider and how well we can integrate with them will impact the risk assessment of the cloud engagement; it is meaningless to have a locked down server inside an IaaS if we can disable all the controls through a weakly secured management plane!

Big bugs have lesser bugs

Sun, 05 Feb 2017 18:57:05 -0500

The Siphonaptera has various versions. The version I learned as a kid goes:

    Big bugs have little bugs,
    Upon their backs to bite 'em,
    And little bugs have lesser bugs,
    and so, ad infinitum.

We make use of this fact a lot in computer security; a breach of the OS can impact the security of the application.

We could even build a simple dependency list:

    The security of the application depends on
    The security of the operating system depends on
    The security of the hypervisor depends on
    The security of the virtualisation environment depends on
    The security of the automation tool...

Of course, in reality the dependency list is a highly branching tree; the app may depend on multiple different things in order to be secure, such as the source code repository, the build environment, the single signon solution…

This seems obvious; if I can break into the central management tool then I can use that to break into VMs and from there impact the app.

Bidirectional

We also have some awareness that the list isn’t one way. We know about remote code execution and privileged escalation attacks, so we can also say:

   The security of the operating system depends on
   The security of the application

A problem comes in when we have a large difference in organisational security boundaries. A broken app allowing privileged access to the OS is annoying, but it’s still (normally) only the app that’s directly impacted.

However when there’s a higher level structures in place then we may have multiple applications impacted; a hypervisor bug may allow an application in one OS to impact a different application in a different OS.

It gets even worse; a bug in the hypervisor management layer (e.g. the VMware agents feeding into VSphere or VCOps) could cause a bug in an application leading to impact every server in the enterprise.

We saw a variation of this, recently. An ansible bug allowed a compromised machine to run commands on the controller node. And, of course, the controller typically has access to every machine managed by ansible, including root access on those machines.

   The security of the application depends on
   The security of the operating system depends on
   The security of the hypervisor depends on
   The security of the automation depends on
   The security of the hypervisor depends on
   The security of the operating system depends on
   The security of the application

With all the branches!

Assessments

When we do application risk assessments we tend to focus on the app and the OS that runs it. We don’t look at the automation environment, the authentication tool, the hypervisor management because these are “the same” across app deployments in an enterprise, so not worth testing hundreds of times over.

This can lead to a gap; we need to ensure these tools are properly covered and managed. If you pay for an external team to pen-test your app then it may be a good thing to resist the temptation to limit their scope; let them try and break the higher level constructs that your app depends on. You might learn something new; something a real attacker could use against you!

At the very least ensure your tooling is fully patched and that bugs found against them are evaluated as having company wide impact.

Make it easy to use

Mon, 23 Jan 2017 19:18:30 -0500

There is a temptation in computer security circles to aim for the perfect. After all, we know that if there is a hole then it will be found and will be exploited. So we tend to build (hopefully! ahem OpenSSL) secure products that will withstand attacks… and then fail at usability

Let’s take a brief travel through…

Unix naming services

NIS

In the long distant past (the 1980s), Sun Microsystems created a system called NIS (Network Information Services)¹. This was a massive step forward for large computer networks; it allowed a central password “map” rather than needing to distribute it to every computer; it provided a central control point (change one file and every client machine picks it up). Today this is expected, but in the 80s it was new and clever.

As time went on, though, we realised things weren’t as nice as we’d like. People like Alex Muffett created fast crack routines. NIS exposed the raw crypt strings and so people could take them and off-line attack and steal passwords. You didn’t even need to have a login to steal this information; just access to the network.

Further, NIS was inherently based on network trust; someone on your network could fake a NIS server and convince your machine to talk to them; now they can login to your machine and become root. There were some small attempts to increase NIS security but, really, it was a security nightmare.

NIS+

So Sun next created NIS+. This solved many of the problems with NIS; servers had keys so the communication could be trusted; data could be protected (no more crypt string exposure); it even allowed for hierarchies of servers, so tree based definitions could be created (eg at the top level we could have SA accounts valid everywhere; the chemistry department could run their own account system and only chemistry servers would know about those accounts).

Nice! Well, not so much.

The problem with NIS+ was that the management of it was horrendous. For a headache, look at the procedure to change the root server’s IP address. There are 9 primary steps, each with multiple sub-steps, including a “wait 12 hours” period. WTF?

Running NIS+ at scale was just too complicated for many organisations. They stuck with NIS. The insecure, broken solution. Because it worked for them.

NIS+ was so bad that by the time Solaris 9 came out, mention of it had been dropped from the official Sun training courses. NIS was still mentioned; LDAP was the primary focus. But NIS+ had been dropped totally.

Compare to Microsoft NT Domains

If you look at the old NT domain model you’ll see a lot of similarities between this and the NIS+ model. After all, they’re trying to solve similar problems. The difference, though, is in usability. Let’s take the ‘change primary server IP address’ problem, from before. With NT this is pretty simple; any replica can be promoted to become a primary. NT cheats; every domain server (primary or replica) shares the same identity, so it’s easy to move the primary around.

NT has its own problems (mostly due to backward compatability - LANMAN hashes!) but it was massively easier to manage.

NIS+ lack of “ease of use” lead to failure for adoption. NT pretty GUIs made it easy to use.²

Today’s fight

The fight, as I type, is the same as last week; WhatsApp.

To summarise, the Grauniad wrote an article about how WhatsApp had a security backdoor. The security world went mad, with claims, counter-claims, rhetoric, demands to opensource the software, or use the Signal software.

Why is this important? Well, WhatsApp is meant to provide E2E (End To End) encryption of messages. Only the sending and receiving devices get to see the plain text. If someone manages to attack anywhere in the path (eg the central server) then they still can’t read the messages. So, clearly, a backdoor that breaks E2E is important.

The reality is that the backdoor isn’t anything of the sort; it’s a usability trade-off. Under specific circumstances (essentially; the receiver gets a new phone and generates a new key) the sender will automatically re-encrypt the message and send it. An alert is provided to the user. With Signal the user is prompted about the change and given the option to re-encrypt.

What is the risk? Only unsent messages are retransmitted. And there’s limited attack points; primarily the central servers and the recipients mobile network (de-register the legitimate device, register your own device, receive messages!). The latter is a potentially legitimate attack vector in some countries, with tight crack-downs on dissent.

The “perfect security” team claim the Signal approach is better, but don’t take into account user experience (“resend? Sure, I know Fred gets a new burner phone every week to avoid security forces”). The WhatsApp team believe that the vaste majority (99%) of rekeys will be due to standard “new phone” purchases.

So here we have a usability vs perfect security trade off.

Conclusion

“100% secure or go home” is a very binary view of the world. Now for a megabank or a government (or a target of a major government) then such a posture might be true. After all, who knows what compute resources the NSA can bring to bear on you :-)

But, really, most people don’t need perfect security; they just need “good enough” security. If you’re in a risky situation then this XKCD is your bigger risk than an attack against a theoretical weakness.

If you do need perfect security in your communication then maybe Signal is the way to go; if you just need ‘good enough’ then WhatsApp will work.

Both are a gazillion times better than SMS, which is what happened before!

As my girlfriend frequently reminds me, don’t let the perfect be the enemy of the good

Originally “Yellow Pages” until British Telecom sued for trademark infringement; this is why the commands still begin yp - e.g. ypcat ^[return]
Yes yes, there are other reasons why NT beat NIS+; desktop integration being a major one. But it was massively easier to manage an NT domain than a NIS+ one. ^[return]

Stop, step back, take a minute, slow down

Mon, 16 Jan 2017 21:52:51 -0500

There’s an old comment; “A Lie Can Travel Halfway Around the World While the Truth Is Putting On Its Shoes”. This came from a pre-internet world. Today a lie can travel around the world in seconds.

Personal annecdote

Last year I broke the MBR on every hard disk on my home server. I was panicing. I really didn’t want to rebuild and restore from backup, and then re-rip all my DVDs; such a time sink! So I stopped; stepped away from the keyboard; got a (decaf) coffee. Took the time to think.

By taking a minute, slowing down, I was able to think about the problem and come to a solution that saved me a tonne of time and effort (and, effectively, zero downtime except for one reboot).

Real world incidents

An incident happens; in American it’s probably a shooting. It could have been the Christmas truck incident in Germany. Or Benghazi. Or 9/11. Immediately twitter and facebook are aflame with theories and stories. But we don’t actually know what’s happening; the facts are still being investigated. Stop; step back, take a minute, slow down. Wait a few days and see what the investigation reports. The truth probably isn’t what you read on social media.

There’s even an infographic…

Infosec stories

Last week the Grauniad wrote an article about how the world’s leading secure communication app, WhatsApp, had a security backdoor The security world went mad, with claims, counter-claims, rhetoric, demands to opensource the software, or use the Signal software.

Stop; step back, take a minute, slow down.

The “day two” story shows that this isn’t a backdoor (which has a specific meaning). It’s a compromise between usability and security. Was it the right compromise? Maybe not. But it’s not a backdoor. The world isn’t ending, the WhatsApp servers can’t read your stuff, the FBI can’t get a warrant to read your messages.

Corporate environments

Your company has just been breached (it will happen). You’ve done a thorough incident response and decided “XYZ technology wasn’t good enough”. Perhaps it was that an SA’s laptop got infected with a key logger (AntiVirus software wasn’t good enough), or a web site leaked credentials (inadequate scanning).

A common response is “XYZ wasn’t good enough; let’s replace it with ABC”.

Stop; step back, take a minute, slow down.

The “day two” story is that what you’ve got is probably as good as what you’re proposing. ABC may have detected this issue, but it may not detect another issue that XYZ has been protecting you from. The problem may actually lay elsewhere (an underlying assumption that you’d repeat with the new technology).

Conclusion

It’s easy to fall into the Politician’s syllogism

We must do something
This is something
Therefore, we must do this.

Whether this is in response to an external event (shooting, bombing), or an internal issue (something broke) or even a personal issue (argument with your Significant Other), there’s a rush to try and fix things.

Stop; step back, take a minute, slow down.

Make sure you’ve got the correct information and the right analysis; the situation may not be what you think it is. By stepping back and taking that extra time you might save yourself a lot of time and effort and avoid replacing a working system with one with new and different oddities.

The Itsy Bitsy Security Spider

Sun, 08 Jan 2017 22:47:50 -0500

Sometimes we can learn a lot from nursery rhymes; I’ve previously shown how A Hole In My Bucket can teach us about understanding problems and how this can lead to security issues.

The Itsy Bitsy Spider can also teach us…

So what is the story of the spider?

Spider starts to do something productive
Event happens that destroys the progress
Spider starts over to do something productive

But what we don’t see, here, is the spider learning from the event. It has fixed target (climbing the water spout) and is focused on achieving that goal.

Now we see similar “event happens” in our lives.

Management change

A common one in large enterprise environments is a change in management. The new hierarchy want to stamp their authority on their department and so will pick on something and force change; they throw away the old and replace it with their favourite technology.

Except technology isn’t the whole of the problem space. Technology is easy; it’s process and controls that are hard.

So the incoming management tree might demand a replacement of product X with product Y. The engineering teams will work out how to get product Y working in the company; they’ll replicate existing data models, process workflows, user interfaces (request/approval/re-certification/etc).

And at the end of the day we’ll have all the existing problems replicated into the new system.

New management will have washed the spider out of the spout, but then engineering climb up it exactly the same as before. Nothing has been learned, nothing has been improved.

External attack

This rainstorm is the one we all dread. We know it will happen¹ and try to limit the consequences.

So we build a solution; we have defence in depth; we have overlapping controls and processes; we have so much stuff running on machines that we chew up more resources monitoring than we do on the actual work of the company. And we still get hacked.

As part of the incident response management goes into a panic, bring in consultants to analyse the existing solution. These consultants will spend months interviewing people and writing reports… that match their existing preconceptions and recommend installing new solution.

We’re back in the same boat as before; technology change but not addressing the real problem.

Conclusion

What we’re really talking about, here, is a variation of “group think”. The company has a set of processes and standards to be followed and any new technology will be forced into that viewpoint. A rainstorm may force a change of technology but we’ll just rebuild in the same pattern.

It’s hard, but sometimes you need to think outside the box and come up with new ideas, rather than just new technology.

Next up: Humpty Dumpty can be put back together again.

You will be hacked. ^[return]

Always Listening Devices

Mon, 02 Jan 2017 13:22:58 -0500

Recently we heard news that the police had requested Alexa recordings to assist with a murder enquiry. The victim had an Amazon Echo and the police feel there’s useful data to be obtained.

This leads to speculation about what sort of information is recorded by these devices, and how secure are they?

What type of devices are we talking about?

There are a number of devices out there these days which you can talk to, to request things. These include Apple’s “Siri”, Google’s nameless “Assistant”, Amazon’s “Alexa” and Microsoft’s “Cortana”.

Some of these require you to press a button to wake them up, but others are always listening for a “wake word”. Here you might say “Hey Siri, what’s the weather?“. The device hears the magic word “Hey Siri” and then works out what you said and acts accordingly.

These devices come in varied form factors; any modern smart phone likely has an assistant which may be voice activated (and may be enabled by default; there may be switches to turn it off or convert it to activate in limited cases); your smart watch may have it (my Motorola 360 will listen for “OK Google” but only when the watch is active; so raise your arm to wake it up and talk to it); your laptop may have it (macOS Sierra has Siri, but requires a button press to activate); tablets may have it; and, of course, the new “hot” items of Amazon Echo and Google Home, which are designed to sit in your house and listen to you.

How do they work?

These devices don’t typically process your voice commands locally. They just don’t have the processing power or storage to understand natural language commands. Instead they are always listening to you for the wake word. These are generally fixed or (as with Samsung’s attempt) trainable. This processing is done locally.

Once the magic word is said then the device will start streaming data to the service provider, along with a small buffer of data from before the command (Amazon claim “a fraction of a second” in their FAQ).

The cloud service now has a copy of your voice recording, can attempt to parse it, and respond accordingly. Because the activity is parsed remotely it can be made smarter and updated and new commands added just by the provider updating their servers; no remote update is needed. New “skills” can be added at any time.

Amazon Echo in practice

I received an Echo as a present so, of course, I started to network sniff to see what it was doing. Excluding DHCP/ARP and other normal local traffic, a few things stood out.

Every 30 seconds I saw a UDP packet of 159 bytes being sent out to some ec2 instance on port 33434. This might be the Amazon Device Messaging (ADM) which implies a remote command might be able to wake up your Echo up.

Every 30 seconds, HTTPS traffic to an Amazon IP address (205.251.242.52 in my sample) of 41 bytes. This would seem to be another polling mechanism.

NTP traffic to various places

Infrequent ping traffic to the default gateway, the DNS server I listed in DHCP and Google and OpenDNS DNS servers, followed by DNS lookups for www.example.org and a HTTP attempt to the result. This is possibly a connectivity attempt to verify the WiFi connection is still working properly.

A small communication with device-metrics-us.amazon.com. Hmm!

And that’s all I saw when the Echo was idle.

When I spoke the activity word then I started to see a lot of https traffic to the same Amazon IP address previously mentioned. So perhaps the 30 seconds polling test mentioned earlier may just be a “is the remote end still there?” test, to allow for quicker sending of data. After the conversation had completed then the traffic resumed back to idle levels.

I also tested streaming of music (“Alexa play 80s pop”). What was interesting, here, was that the song appears to be buffered locally; I could see a LOT of https traffic (from a cloudfront server) which then stopped. So it’s clear the Echo has enough memory to store a 4 minute song locally.

What are the risks?

Based on the above data, the risk would appear to be small. Let’s look at it:

Amazon etc store your voice recordings

They do this to help train your device to your voice (or so they claim). In theory this now creates a record of all your requests and so could become subject to a subpoena. Even if you trust Amazon/Google/whoever will they fight for you to keep this data out of the hands of the police?

How much do they send before the magic word?

We only have their word that a fraction of a second is sent prior to the magic word. They might lie.

Activity may change

What I record is how Echo works today. I have no doubt that Amazon can force a firmware update to devices as necessary. So some of the protections we see (e.g. the LED ring lighting up to show streaming) might disappear. Could the Feds demand a version of the firmware that would record and stream permanently without a visual indicator? How many of the protections are software and how many are hardwired?

Attacks against the device

A simple nmap of the device shows no listening ports, so an attacker on your local network (remotely by a router exploit!) do not appear to have an attack vector this way. This leaves custom skills. Now these are processed remotely on a remote server; the amount of data sent back is primarily limited to a “speech” text channel or to an audio channel. If there was a bug in the codec or text-to-speech modules then an attacker might gain access that way, but it requires you to have enabled their skill.

Other risks

Attacks against the account

This, in my mind, is more likely… your voice recordings and enabled skills are all controled via the Alexa site, and this is protected by your normal Amazon password. Is that secure? The data you’re now protecting has extended beyond your purchase history and ability to buy stuff, but to voice recordings and command histories.

Inappropriate use

There’s a second class of risk relating to inappropriate use of these types of device (why would you link it to your bank account when there are no voice level controls? Anyone in your house could ask for the same information). This includes the built-in “buy stuff from Amazon” skill, so turn that off! But that’s another blog post :-)

Conclusion

Everyone has to make their own risk assessment of these devices. Based on what I’ve observed and tested, I don’t think the Amazon Echo creates a large exposure. Although the device may be “always listening” it doesn’t send data until the magic word is spoken. I’ve seen a twitter joke about Orwell’s 1984 along the lines of “we didn’t need to have enforced surveillance, we brought it into our own homes”, but I don’t think this is accurate. These devices aren’t streaming 24x7, but only on demand.

So why did the police demand recordings in the murder case? I don’t know, but what if the victim had said “Alexa, Fred is killing me!” while he was being murdered? That would be recorded and could be relevant information :-)

Personally, until I find out otherwise, I feel the Echo device is not a large risk… as long as I don’t connect it to personal data sources (no, Alexa, you won’t get access to my bank account!).

But what about laptops?

A common recommendation for laptop owners is to cover your camera (and block the microphone) when you’re not using them. You can buy camera shutters that can be stuck over the camera, allowing you to slide open/closed as needed. Why is this different to an “always listening” device?

It’s a matter of attack vectors. Earlier I showed that there’s no open ports on an Echo; an attacker can’t break into it so easily. However a laptop is possibly running Windows or macOS, is used to read emails or surf the web and is at a much greater risk of malware being installed which could then access the camera or microphone. The Echo, being an appliance device, has fewer malware insertion vectors (I won’t say none because everything has bugs) and so doesn’t present the same risk surface.

SSH keeps disconnecting

Sat, 24 Dec 2016 21:16:52 -0500

This blog post is of a more practical nature, and may be of use for people at home who ssh into servers and then come back later to find their session disconnected. It might also help some people in offices with nasty firewalls!

Basically the scenario goes something like:

ssh into a server
lock your screen, go away for a few hours
come back, unlock your screen
ssh session has been disconnected

So how does this happen, and what can we do to stop it?

Network Address Translation (NAT)

The first thing we need to be aware of is how most home router work. When you connect to a standard home consumer ISP they give you one IP address¹.

Now every device you have including the router needs an IP address. Your desktop, your laptop, your cellphone, your games console, your iPad… if you go around your house and look at the number of devices you’ve connected to your WiFi (or plugged in with a cable into the router) then you might be surprised. I counted at least 20 devices in my house; there may be more!

So the router needs to work hard in how it lets all those devices share the same IP address. It does this by creating a local network on your WiFi/LAN and devices get their addresses from that. So your laptop may get 192.168.1.10, your cellphone might get 192.168.1.11, your iPad maybe 192.168.1.24 and so on.

Where this gets clever is that the router then “changes” the address whenever any of your devices talks to the internet. So lets say your iPad hits this web site. Your iPad is 192.168.1.24 and wants to talk to my server. When the traffic gets to the router it will translate the first address into the external address provided by your ISP. So now your iPad can talk to my server.

The problem is that the router needs to remember that it’s done this, so that when my server replies then it knows to send the response back to the iPad and not to your desktop or your phone. The router needs to keep a connection table in memory of all the address translations it is doing.

Dropping connections

There are a number of problems with NAT, but two are relevant here:

The memory in a router is small; if too many connections are made then it might have to drop some traffic (I saw this with the older Verizon FIOS ActionTec routers; it only had a small NAT table).
Connections might break messily; the router needs to have a way of cleaning up the connection table entries for dead connections.

This is normally solved by having a timer associated with each entry; “when did we last see a packet?”. If the table gets full and a new entry needs to be created then the oldest connection could be dropped and closed down. Similarly if the connection is too old (“we haven’t seen a packet for over 2 hours”) then the router might consider this a dead connection and drop it. Some routers are even more aggressive and may drop connections after 30 minutes or even sooner!

This second case is where we start to see idle sessions get dropped. Because the ssh session hasn’t sent any traffic in a few hours the router thinks it is a dead session and will close it.

KeepAlive

The standard workaround for this sort of problem is to make sure the router never thinks your connection is idle; send “fake traffic” at regular intervals. In computer jargon this is called “keep alive” traffic; we’re sending it just to keep the connection alive.

Now the first thing someone notices when reading the ssh manual page is something called TCPKeepAlive. Hey, that sounds good! It turns on keep alive packets at the TCP layer! Unfortunately… not. TCP KeepAlive packets weren’t designed to handle intermediate router drops; they’re more there to keep the local socket alive and detect if the remote server has gone away (and so drop the connection). The traffic is frequently sent at a much lower frequency than the router needs, and so it doesn’t prevent the connection being dropped.

The ssh authors noticed this and in version 2 of the protocol (there’s a 99.99% chance you are using SSH2; SSH1 has known security issues and has been turned off almost everywhere) they added a “No Operation” (NOOP) message type. This can be used as a simple Keep Alive packet.

How you turn this on depends on your client.

Unix (and MacOS)

In your $HOME/.ssh/config file you can turn on KeepAlive with the following commands:

Host *
  ServerAliveInterval 600

This tells the ssh client that if it hasn’t received a message from the server within 600 seconds then send a NOOP message to it, to get a response. This works pretty well at making the router think the connection is still busy and so don’t drop it as idle. Nicely this packet is only sent if you are idle, so it doesn’t cause any extra traffic on active sessions.

Depending on how aggressive your router is, you may need a smaller number.

Windows, using putty

putty is a pretty good Windows ssh client. It’s the one I use when forced to use windows! It also has the ability to send keepalives, which can be found on the Connection section of the connection menu:

You can enter a timeout here, and save it as part of your connection profile.

Others

If you have a favourite ssh client with keepalive ability built in then feel free to drop me a note in the comments!

Conclusion

NAT is nasty and some of the compromises it causes can cause longer running idle sessions to break. This doesn’t really impact most users because typical activity (eg web surfing) uses short-lived connections. But for long lived (eg ssh) sessions that can be idle for a long time then we may see disruption.

We can work around this at the application level by sending application layer keepalive traffic. Many ssh clients have this ability built in.

Connection dropping isn’t solely caused by NAT, but it is a common cause, especially for people initiating connections from home with an ISP provided router.

Not always, but if you’ve requested more than one address then you should know enough to realise this is a simplification. ^[return]

Backup and restore

Sun, 11 Dec 2016 20:44:25 -0500

Have you tested your backups recently?

I’m sure you’ve heard that phrase before. And then thought “Hmm, yeah, I should do that”. If you remember, you’ll stick a tape in the drive and fire up your software, and restore a dozen files to a temporary location. Success! You’ve proven your backups can be recovered.

Or have you?

What would you do if your server was destroyed? Do you require specialist software to recover that backup? Do you still have the install media? The license keys?

What would you do if the datacenter caught fire and you had to rebuild from scratch? (Assuming you ship your backups offsite…)

Or had everything encrypted by malware…

SF MTA

In November 2016 the San Francisco Municipal Transportation Agency got hit by ransomware. It encrypted a lot of their files. They were unable to run the terminals to allow passengers to buy tickets, and were forced to allow free travel.

Now in many of these cases the victim ends up paying up to get the decryption keys to recover their data. But the SF MTA didn’t do that; they went to their backup tapes and restored everything they needed.

Document, test

When you are in a disaster recovery situation you need to have standard processes and procedures fully documented. How do you restore data? How do you recall tapes back from offsite? Do you know what tapes to recall (hint: the tape inventory may have been encrypted!).

Simple procedures that you test are critical. Assume a worst case scenario and work out how you would recover from that. Note problems that arise during the tests and update your procedures.

Protect the backups

Many backup systems leave data “on line”, especially those that backup to a remote disk. If these backups can be modified then they could also be encrypted. A backup is useless if you can’t read the data.

These backups contain pretty much all your data. They should be treated as highly confidential data. If you have PII (Personally Identifiable Information) in that backup then you need to treat it the same as any other PII data. This may mean it needs to be encrypted… in which case you now have encryption key management problems (not so “simple procedures”).

You also need to be careful where and how you store your backups; again in November Michael Page, a UK based recruiting firm, had their data leaked. These were SQL backups from a service run by Capgemini. Somehow production quality data was placed on a development server, and exposed.

Personal anecdote

I have a very simple process for my backups; it may not be efficient, it may waste disk space… but it’s simple.

I use the native OS backup software. So for ext[234] disks I use the dump command; for xfs disks I use xfsdump (and when I ran SunOS or Solaris I used their dump commands).

On a Sunday I run a level 0 backup; that’s everything. On Monday I do a level 1; on Tuesday a level 2… so each day has an increment on the previous day.

These backups are then ‘rsync’d offsite to a second server. If the worst happens then I could build a machine and boot it from a DVD and then restore all the data over the network back to the disks (no special software dependencies! It’s part of the standard OS). I have a simple process; restore the latest level 0 then all the incrementals.

I got to do this for real, once. I was in Newark Airport, ssh’d home to read my mail, when I hard crashed the server (known issue; if I ran X twice then it hard crashed). Damn; I wasn’t going to be home for 2 weeks! All my mail and stuff would queue up on my external machines, but… grump

I had a brain-wave; I had a second machine where I was playing around with virtualization… could I recover the broken machine into a VM from the backups? I fired up a new VM, custom configuration for disk sizes and then restored the data. Got on the plane while the restore was running. When I got to the hotel I finished up the process and rebooted… and it worked!

Indeed it worked so well that when I finally got home and rebooted the physical server all I did was transfer over the few files that had changed since the backup run and then shut it down. I’ve been running via VM ever since. The old physical machine is sitting under a table, just in case I need hardware in an emergency :-)

Conclusion

Backups are no good if they can’t be used to restore your servers back to operational state. This means test your processes, including any steps necessary to get your backup/restore software working again.

Protect your backups, both from tampering and from being leaked.

Don’t expect everything to work first time; and don’t be surprised if restores take longer than you expect! (One off-site DR test my company ran ended up with the windows server restores taking 2 days when they only planned for 4 hours; good lesson to learn!)

Using Letsencrypt for TLS

Sun, 04 Dec 2016 14:42:33 -0500

In previous posts I pointed out why TLS is important, how to configure Apache to score an A+ and how to tune HTTP headers. All this is dependent on getting an SSL cert.

Some jargon explained

Before we delve into a “how to”, some basic jargon should be explained:

SSL/TLS

TLS (“Transport Layer Security”) is the successor to SSL (“Secure Socket Layer”). SSL was created by Netscape in the mid 90s (I remember installing “Netscape Commerce Server” in 1996). When you hear people talk about SSL and TLS they pretty much use the terms interchangeably. You can think of it as a progression SSL2, SSL3, TLS1.0, TLS1.1, TLS1.2 with each version better than the previous.

SSL/TLS is what makes the connection to the server secure.

SSL Certificate

This is a file you place on your server and this basically says “I am server x.y.z”. The server for this website says “I am server www.sweharris.org”. There are other things this certificate can say (”I can also be called sweharris.org”). Different certificates can also be used for other reasons (e.g. signing code), but I won’t go into that here.

The SSL certificate is signed by a trusted party (the “Certificate Authority” - CA), so when your browser goes to a web site it can verify the certificate is valid. Your browser has a list of CAs that the browser creator trusts. If I tried to create a certificate for “google.com” then no CA would sign it, and your browser would say “Nope!”

Domain Validation

In order to get a certificate signed the CA needs to know that you are who you claim you are. “Domain Validation” (DV) is the weakest and easiest method of validating ownership. This may be done by sending an email to a standard address (eg postmaster or hostmaster). It might be done by requiring you to put a specific file on the web server. Or changing a DNS entry… the idea is to require the requester to prove they have access to a controlled resource associated with the domain.

This techique is used elsewhere; e.g. if you put your website into Google Webmaster Tools then it will require you to put a specific entry on your server (eg /googleSOMESTRING.html). If you want to use Google Apps for your mail then it may want a DNS entry (eg googleSOMESTRING IN CNAME google.com.). These are both “domain validation” examples.

Extended Verification

Extended Validation (EV) certs take a deeper approach to validating the request; they also check the owner claims. So, for example, I might be able to get a DV cert for chas3.com, but I would not be able to get an EV cert saying “this is owned by JPMorgan Chase”. An EV cert is meant to help reduce phishing attacks by helping the web user spot if an EV cert is in use.

Compare my site to Chase’s:

We both have the green padlock, showing the cert’s are good, but the Chase entry has more details about the owner next to the padlock.

For a site like mine a DV cert is sufficient. You need to decide if you want a DV or EV for your site.

Getting a cheap cert

When I first started out in 1996 getting a cert was a hard tedious process. We had to fax (fax!) details to one of the very few CAs that were around, and it took weeks to get the cert. Painful. It was effectively an EV process, long before such a thing really existed.

As time went on the DV process became easier to get, but they still cost money. It wasn’t really worth it for most people.

In 2012 I noticed that StartCom were providing free DV certs, under the brand StartSSL. This was an Israeli company and their CA was a common one. It made sense.

The StartSSl process was pretty painful. It was totally manual and involved using a web site with cut’n’paste. The DV process expired after 30 days so every renewal required revalidation. But it was free, and I only had 7 certs with them, so I could jump through the hoops every so often.

Unfortunately, it’s not clear if StartSSL is still trustworth, and may now operate from China.

Letsencrypt

Fortunately Letsencrypt started up. Their goal was to automate a lot of the manual steps, while still providing free DV certs. They even provide a client so you can basically say “encrypt my site” and their software will create the necessary cert files, do the DV process (stick a file on the server), retrieve the signed result, reconfigure your webserver daemon and, boom, in seconds your site is now SSL enabled. The certificate is only valid for 90 days, but that’s not a problem; the same software can automatically renew the cert before it expires.

Very impressive.

Unfortuntately it doesn’t work for me. My site is replicated across multiple providers (one copy at Linode in Texas; one copy at Panix in New York; one copy at OVH in Canada). I manage content on my home machine and rsync the results out. I also manage the webserver configurations via ansible.

If I ran their client on one machine then I’d have to copy the resulting certs to the other two… it’s doable, but messy.

However there was another way of doing the DV certification, and that’s via DNS. Now my DNS is also replicated (hidden primary server at home; public primary at the three sites). So this means I could do the cert DV validation via DNS from my home machine, then push the resulting cert out to the web servers via ansible.

The original Letsencrypt client couldn’t do DNS validation, but Lukas Schaur created an alternative shell based client called deydrated. This is a much simpler program, but allowed DNS mode to be used, and had a call out so I could modify how the DNS change ran.

How it works

Last week one of my certs needed renewing, so I went through the renew process:

$ cd etc/localnet/Letsencrypt/
$ ./refresh
 + Valid till Dec  6 23:35:00 2016 GMT (Less than 8 days). Renewing!
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting challenge for linode.spuddy.org...
 + Responding to challenge for linode.spuddy.org...
 + Challenge is valid!
 + Requesting certificate...
 + Done!
 + Creating fullchain.pem...
/home/sweh/etc/localnet/PlayBooks/Files/Postfix/Per-host/linode

The refresh command calls ./dehydrated -c to do all the hard work of talking to the Letsencrypt servers. It noticed that the linode cert needed renewing, and got a new signed cert. My script then compared all the managed certs and copied any changed ones into the ansible tree. In this case the single cert for the linode Postfix server (also used by INN) was updated.

Since a change was done I could then use ansible to push it out:

$ cd ../PlayBooks/
$ ansible-playbook -l linode postfix-inn.yml 

PLAY [Set up postfix config on mail servers] ********************************** 

[...]

TASK: [send machine specific certs] ******************************************* 
changed: [linode] => (item=privkey.pem)
changed: [linode] => (item=fullchain.pem)

NOTIFIED: [restart postfix] *************************************************** 
changed: [linode]

PLAY [Set up INN] ************************************************************* 

[...]

TASK: [send INN privkey] ****************************************************** 
changed: [linode]

TASK: [send INN cert chain] *************************************************** 
changed: [linode]

[...]

PLAY RECAP ******************************************************************** 
linode                     : ok=21   changed=4    unreachable=0    failed=0

And that’s it! I can also verify the new certs are in place and working

$ cd ../Letsencrypt
$ ./check_remote nntp linode.spuddy.org
             linode.spuddy.org NNTP: OK notAfter=Feb 27 16:31:00 2017 GMT
$ ./check_remote smtp linode.openvpn linode.spuddy.org
             linode.spuddy.org SMTP: OK notAfter=Feb 27 16:31:00 2017 GMT

How to know when to renew a cert

It would be bad to allow a cert to expire, so you need ample warning of when a cert needs to be refreshed. Because of the tree used by the dehydrated code is consistent this is very easy to script:

#!/bin/ksh -p

# Warn if cert will expire in this many days
let DAYS=${DAYS:-7}
let warn=DAYS*86400

typeset -R25 h
bad=""

cd /home/sweh/etc/localnet/Letsencrypt/certs || exit
for a in */cert.pem
do
  b=$(openssl x509 -checkend $warn -in $a)
  if [ "x$b" = "xCertificate will expire" ]
  then
    b=$(openssl x509 -noout -dates -in $a | sed -n 's/^notAfter=//p')
    h=${a%/*}
    bad="$bad\nletsencrypt $h: $b"
  fi
done

if [ -n "$bad" ]
then
  print Following certificate expiration dates:
  print "$bad"
fi

So I can now see what certs are due to expire:

$ checkssl
$ DAYS=20 checkssl
$ DAYS=25 checkssl
Following certificate expiration dates:

letsencrypt           bofh.spuddy.org: Dec 28 23:46:00 2016 GMT
letsencrypt        gallery.spuddy.org: Dec 28 23:47:00 2016 GMT
letsencrypt          games.spuddy.org: Dec 28 23:47:00 2016 GMT
letsencrypt           home.spuddy.org: Dec 28 23:47:00 2016 GMT
letsencrypt       mercury7.spuddy.org: Dec 28 23:48:00 2016 GMT
letsencrypt     panix-news.spuddy.org: Dec 28 23:48:00 2016 GMT

This is easy to put into a cron job. Indeed

1 1 * * * /home/sweh/bin/checkssl

In the morning I can refresh any certs necessary while drinking my coffee :-)

Automation

So far I’ve only automated 90% of the job. I still have to run the refresh and ansible-playbook commands. Partly this is because it’s so easy, partly because it gives me positive feedback (I know the work was done and the push successful), but mostly because this is “good enough” for my use case.

When I started using this process I would occassionally get errors with the DV step; I’m not sure if that was due to DNS propagation issues or caching at the LetsEncrypt side or something else. I haven’t had a problem for quite a while now, but I like to keep my eye on it.

But we can see, quite easily, how a daily “refresh/push” process could be created so the overnight “checkssl” job could become a “check and refresh” job.

Currently I have 12 certificates managed this way, and it takes less than 5 minutes a month to manage.

Summary

Where DV certs are sufficient, a tooling around Letsencrypt is perfectly adequate and sensible. It can even be used for internal web sites, as long as DNS challenge/response can be handled.

There’s little excuse to not use SSL on web sites any more!

LXD and machine containers

Sun, 27 Nov 2016 16:31:00 -0500

A few months back I was invited to an RFG Exchange Rounds taping, on containers. There were a number of big name vendors there. I got invited as an end user with opinions :-)

The published segment is on youtube under the RFG Exchange channel.

Unknown to me, Mark Shuttleworth (Canonical, Ubuntu) was a “headline act” at this taping and I got to hear some of what he had to say, in particular around the Ubuntu “LXD” implementation of containers. I’m not convinced he’s going in the right direction and I pulled what my girlfriend calls “sceptical face”.

Beyond the problem of “machine containers” (which are, basically, the “container as a VM” model, and which is specifically promoted by LXD), I have a real problem with how he sees machine containers being used. This is typified by the segment starting at 2 minutes; we can take an 1990’s Linux install and run it unchanged inside a machine container. It’s very possible that this would work. I ran old 90’s compiled executables for many years; the Linux kernel is pretty good at backwards compatibility in this way. (I doubt a real old a.out format system would run, but it’s likely an ELF system should… albeit with some kernel warnings about deprecated syscalls).

The discussion (which doesn’t appear to be in this video, unless I’ve missed it) also touched on “run anything”, with Mark saying that, absolutely, this is a supported use case.

The problem is “just because you can, doesn’t mean you should”. The “run anything” model leads to Wild West deployments.

In an enterprise environment we spend many many hours in standardising our setup. We force a standardised operating system (RedHat or Ubuntu LTS or SUSE, typically, for stability). We force a common identity and access management (I&AM) solution. We have a standardised software stack deployed as part of the build (monitoring, backup, scanning, privileged control…).

We don’t want people running anything they like. I don’t want the introduction of Debian installs inside a machine container inside my RedHat or Ubuntu environment. Yes, LXD supports this; RedHat modified Docker supports this; I could even wrapper generic Docker to do this. But this way leads to insecurity.

Software delivery

I was at a cloud forum last week and a software vendor asked “do you want me to provide my software in the form of a container”.

No, really, I don’t. If this container allows for access to the OS (eg via SSH) then I need to control that access, which means it’s got to be integrated with my I&AM processes and with my monitoring processes. If there’s persistent data then I need to back it up. The whole of the control stack that I’ve optimised for my chosen OS needs to run in that container.

The only time I would accept a containerized application is if it was a true appliance. An immutable image that is configured via external data (eg from the docker run command line) to point to external resources. Now a container may be a suitable deployment mechanism.

But don’t give me a full machine container and expect me to run it as a virtual machine. I never accepted VMware images for this reason; containers don’t change the security stance.

Conclusion

As I’ve written before, machine containers are perfectly valid use of container technology. But if you treat a container as a VM then you must run it as a VM. This means full integration into your control stack. You don’t (shouldn’t!) let anyone start up a VM on your network with any OS they like in it; you shouldn’t let anyone start a machine container with untrusted unknown content.

Building my home server

Sun, 27 Nov 2016 13:06:29 -0500

A couple of weeks back I got a new case for my PC. Previously I was using a generic mini-tower and then had an external 8-disk tower (Sans Digital TR8MB) connected via an eSATA concentrator (4 disks per channel).

It’s been working OK for years but every so often the controller would reset (especially under write loads); no data lost but annoying. Also after a power reset (eg a failure, or maintenace) then frequently one or two disks (slot 2 in both halves!!) weren’t always detected and needed reseating and re-adding to the RAID6 (yay for write-intent bitmaps, so recovery is quick!).

I spotted the Zalman MS800 tower. This has 10 bays. Not enough; I have 12x3.5 disks and 2x2.5 SSDs. But I’d previously bought some 5.25->3.5 adapters that let you put 4x3.5 disk into 3x5.25 space (1 was using one of them in the old case, and the other 2 were emergency backups incase the external tower broke).

So 9 of the slots could fit 12 disks and the 10th slot could contain both SSDs. And, hey, it was only $60 (down to $40 if I can be bothered to fill in the rebate card).

This would also mean I needed some better controllers; the old SiL eSATA card that came with the old disk tower wouldn’t work for the internal disks, so I bought 2 LSI 9211-8 cards; that’d give me 16 channels; more than enough (in theory I could have just used one card, the motherboard has 5*SATA-3Gbps ports which might have been suitable). My existing 650W PSU should be sufficient to handle all of this…

Quite amazingly it worked! I needed to fiddle with the boot stuff a fair bit but I think it’ll now boot unattended if necessary :-)

Speeds are faster than before. Now I get (using hdparm -t)

         SSD: 375MB/sec  (I got 226 using internal ports)
2Tbyte disks: 133MB/sec
4Tbyte disks: 145MB/sec  (I was getting 125 using the old eSATA card)

The raw disk speeds don’t look that much better (in particular the 2Tbyte disks don’t show any real change). But what about RAID performance?

The two SSDs  are in a RAID 1  and get 375 MB/sec
The 4*2Tbytes are in a RAID 10 and get 267 MB/sec
The 8*4Tbytes are in a RAID 6  and get 532 MB/sec

Heh, we can definitely see the benefit of load distribution for reads! This last number is much higher than the old eSATA card because there’s no contention on the port concentrator. Big win! And that’s what I’d hoped for.

Each of the disk adapters has its own fan (120mm). The case has two fans (there’s also a way of adding a fan internally if necessary, but I didn’t use that).

So how hot does this monster run? Most of the time it’s idle-ish, just running a couple of VMs plus “management” stuff (DNS, DHCP, NTP etc).

LM-sensors tells me:

Core 0:      +39.0 C  (high = +83.0 C, crit = +99.0 C)
Core 1:      +35.0 C  (high = +83.0 C, crit = +99.0 C)
Core 2:      +39.0 C  (high = +83.0 C, crit = +99.0 C)
Core 3:      +34.0 C  (high = +83.0 C, crit = +99.0 C)

Well, that’s maybe a little high for an idling machine but it’s well within tolerance. I’m happy with that.

The disks are interesting; the SMART output:

/dev/sda: 194 Temperature_Celsius ... 30
/dev/sdb: 194 Temperature_Celsius ... 30
/dev/sdc: 194 Temperature_Celsius ... 27
/dev/sdd: 194 Temperature_Celsius ... 28
/dev/sde: 194 Temperature_Celsius ... 28
/dev/sdf: 194 Temperature_Celsius ... 28
/dev/sdg: 194 Temperature_Celsius ... 27
/dev/sdh: 194 Temperature_Celsius ... 28
/dev/sdi: 194 Temperature_Celsius ... 28
/dev/sdj: 194 Temperature_Celsius ... 28
/dev/sdk: 194 Temperature_Celsius ... 28
/dev/sdl: 194 Temperature_Celsius ... 27
/dev/sdm: 194 Temperature_Celsius ... 28
/dev/sdn: 194 Temperature_Celsius ... 28
/dev/sdo: 194 Temperature_Celsius ... 40

The two SSDs run slightly hotter than the spinning rust! And the last disk (sdo) is a 1Tbyte disk sitting in an external cradle and just exposed to ambient air for cooling.

It’s quite impressive how much difference fans can make :-) The external disk is hotter than the internal disks, and even more fun the spinning rust runs cooler than the SSDs; the SSDS don’t have any forced air-flow whereas the disks have those 120mm fans drawing air over them.

(Heh, the UPS also claims power load is now down to 23%; previously it was 25% with the external drive bay).

Intel Clear Containers

Mon, 21 Nov 2016 20:11:13 -0500

Containers aren’t secure… but neither are VMs

An argument I sometimes here is that large companies (especially financial companies) can’t deploy containers to production because they’re too risky. The people making this argument focus on the fact that the Linux kernel is only a software segregation of resources. They compare this to virtual machines, where the CPU can enforce segregation (eg with VT-x).

I’m not convinced they’re right. It sounds very very similar to the arguments about VMs a decade ago. It also misses out on the hypervisor, itself, being software. This hypervisor runs in privileged mode and is accessible from the client (eg network drivers, block device drivers, paravirt drivers). VT-x instructions can help with some low level primitives and segregation, but even that’s not perfect (oops, rowhammer!).

Will there be bugs in the Linux kernel that’ll enable one container to break out? Sure! But they’ll be found and patched. And hypervisors aren’t bug free. As of time of writing there’s yet another critical Xen bug meaning one of my public VMs will need to be rebooted as my service provider patches their servers.

Intel Clear Containers

So Intel took this concern and ran with it (pdf).

Essentially they created a “one container per VM” solution and optimised the VM. The hypervisor layer is a stripped down and optimised KVM/QEMU solution. Each VM then runs on a minimal Linux OS (“Clear Linux”) that’s been optimised for this work (fast boot times, minimal drivers, optimised for running a container).

So now we have VT-x protections between containers, because there are no containers sharing the same kernel. To make this solution more transparent they have created a “docker runtime” plugin, so Clear Containers can be managed as if they were normal docker containers (eg Swarm, Kubernettes).

They’ve also worked with CoreOS and the OCI for compatibility with those layers.

But is this necessary?

Many of the risks of a shared kernel can be mitigated simply by not running containers as root (eg unprivileged Docker containers; the default) or by running a PaaS (eg CloudFoundry or Apprenda).

Clear Containers are more heavy-weight than a pure container-in-shared-kernel model and wouldn’t be a suitable replacement for low level container calls (eg clone() with CLONE_NEWPID and similar flags). But they’re faster than building a traditional VM.

In this respect they may be a competitor for Canonical’s LXD solution, or perhaps VMware’s Instant Clone. Intel Clear Containters allow you to create a longer running container solutions packaged inside a docker container wrapper and gaining the protections of VT-x via the “one container per VM” model.

I’m not convinced there’s a good use case… but then I’m not convinced by the “container as lightweight VM” use case, either :-)

Technical Debt

Sun, 13 Nov 2016 21:22:33 -0500

My home server

I was doing an upgrade on my home “server” today, and it made me realise that design choices I’d made 10 years still impact how I build this machine today.

In 2005 I got 3*300Gb Maxtor drives. I ran them in a RAID 5; that gave me 600Gb of usable space. It worked well.

In 2007 I upgraded the machine to 500Gb disks. This required additional SATA controllers, so I got enough to allow new and old disks to be plugged in at the same time (cables galore). I copied the data over. But the “architecture” was roughly the same.

I bought a new PC, moved the disks. I got an external disk array (8 bays) using an eSata concentrator. Switched to 5*1Tb disks. Replaced the PC (quad core!), upgraded the memory, switch to 2Tb disks, switch to 4Tb disks.

Today I bought a new case that could handle 10 5.25” drives, and put in 3 “4*3.25 bays in a 3 bay space” in them. Used some SAS controllers.

So today I have 4*2Tb in a raid 10, 8*4Tb in a raid 6 and 2*500Gb SSD in raid 1.

But the design choice was set 10 years ago. Just because I picked Linux mdraid, and KVM for virtualisation a few years later.

This means I can’t easily switch to another OS base. I’d actually like to try out SmartOS because it has Solaris based ZFS, which strikes me as a lot better way to handle this level of storage. But how do I migrate 24Tb of data, without buying another 8 disks?

That’s technical debt; I could overcome this buy spending money (basically buy a whole new machine, with all the disks and controllers). It might even be a good thing (I could upgrade from my old 2010 based Core i5 750 machine to something more modern). But it’s a lot of money.

Instead I’ve been spending smaller chunks of money to do incremental upgrades of my existing machine. Most likely my next upgrade will be a new motherboard, memory, CPU… which will go into the existing case with the existing disks and existing controllers. I can’t easily switch to a newer better solution.

Software development

So what does all the above have to do with software development?

We also have technical debt here; we’ve built processes and procedures. We may have switched from waterfall to agile (and done it badly, abusing the “scrum” approach because the larger organisation is still waterfall oriented). We’ve added test driven design methodologies. We’ve done peer programming, extreme programming…

But they all focus on small aspects of the overall picture. We’re still looking at deploying operating systems, databases, software. We may have support and change management constraints.

This technical debt is stopping us from using new technologies better; in particular cloud computing.

The number of times I’ve heard lift and shift when talking about moving to the cloud is scary. All you’re doing, here, is “outsourcing the datacenter”. You’re not making use of new technologies, you’re just replicating existing structures. This means you’re going to need to have a DR solution, HA data replication, testing, failover. You’re still needing to patch systems, have reboot and maintenance windows. People still login and su/sudo.

This isn’t cloud computing; this is traditional computing on virtual servers stored off premise.

Just like my “duplicate and rebuild from scratch” home server we may need to do the same with our software. Build with a 12 factor focus. Design for failure. Scale horizontally. Design for hands-off compute, automated deployment, stateless applications, attached storage.

Now we can start to make use of the new technologies. We don’t need a DR solution because the design, itself, is resilient. We don’t do failover testing because there is no failover. We don’t need to patch the OS; we redeploy…

Summary

There’s a lot of benefits to be gained from re-factoring your app, your infrastructure. However getting there isn’t necessarily cheap. We’ve a tonne of historical restrictions to overcome (design, architecture, process, support). We could spend a whole year rewriting our code to the new paradigm, which is time not being spent improving the product.

Can you afford to do this? Can you afford not to do this?

Docker in production

Sun, 06 Nov 2016 15:31:12 -0500

In previous posts, and even at Cloud Expo, I’ve been pushing the idea that it’s the process that matters, not the technology when it comes to container security. I’ve tried not to make claims that at tied to a specific solution, although I have made a few posts using docker as a basis.

I was recently asked my thoughts on Docker in Production: A History of Failure .

Basically they can be boiled down to “it’s new; if you ride the bleeding edge you might get cut”. Which we all know.

There’s some lessons we can take away from that rant, though. Indeed some of the mistakes and lessons come out in the comments. None of them are security related, but they all have impact on the resiliency of the deployments.

Quickly moving technology may change in incompatible ways

One reason for the popularity of RedHat Enterprise Linux and for Ubuntu LTS in large companies is simply due to the compatibility and stability of the releases. You can stick with RedHat 6 for 7 years and not worry (too much!) about patches and enhancements breaking things. Docker, as a technology, is closer to Fedora than RHEL. It’s evolving and changing and improving very quickly. Each release is getting better… but we don’t have compatibility guarantees.

Tooling doesn’t always exist

Again, this is new technology. You may need to invent and create stuff, especially for you local specific requirements. There’s nothing specific to Docker, here. Systems engineering has always had a “invent stuff” role. That’s the fun and why I work in technology infrastructure teams, rather than doing application development.

Containers aren’t meant to be persistent… but can be!

In an ideal world a container based application should not have persistency and treat storage as a service to be attached. But… as previously written, you can attach persistent storage to your containers. Docker even produces a number of options. You can make a database run inside a docker container, but should you?

Don’t blindly sync repositories

You might think that synchronising an external repo to something internal is a good thing; after all, it reduces dependencies on external services and means your build process can continue even if the remote service is unavailable. Well… yes, but that’s not the whole story. Your internal repo should be curated. Don’t just blindly sync trees. The “npm leftpad” issue, the docker signing key issue… both of these would propagate into an internal server if you blindly sync. You also haven’t added any security or repeatability to your process

Not all tasks are suitable for containers

If your server is running 100% on your application then putting it into a container won’t let you run on that machine. If your application hasn’t been developed for horizontal scalability then spinning up 10 containers won’t give you more capacity.

Conclusion

It may not make sense to run your database server in a docker world; it may not make sense to run your low-latency app in a container (the people I’ve worked with won’t even use VMs; it must be physical with specific CPUs with known cache-coherency properties; there must be no disk I/O etc etc); it may not make sense to run your LDAP infra that maxes out all CPU cores in a container…

Containers should only be one tool in the toolbox; it’s not the be-all solution.

Personally, I like the PaaS approach to containers (eg Cloud Foundry or Apprenda); this can hide a lot of the rawness and can provide a more friendly solution… but at the expense of flexibility. It can encourage the use of 12 factor apps, horizontal scalability and use of attached services.

But if you do want to use Docker, then go into it with your eyes open; be aware of the risks; be aware that the technology is changing quickly.

And I’ll leave you with a “Downfall” parody, in which Hitler talks about using Docker in production.

Using SSH certificates

Sun, 30 Oct 2016 09:31:20 -0400

In previous articles I’ve explained how to use traditional SSH keys and why connecting to a wrong server could expose your password. I was reminded of a newer form of authentication supported by OpenSSH; CA keys.

The CA key model is closer to how SSL certs work; you have an authority that is trusted by the servers and clients, and a set of signed keys.

Creating the CA key

Creating a certificate authority key is pretty much the same as creating any other key

$ mkdir ssh-ca
$ cd ssh-ca
$ ssh-keygen -f server_ca
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in server_ca.
Your public key has been saved in server_ca.pub.
The key fingerprint is:
04:f2:60:0a:89:c0:66:2d:1c:a9:da:ca:11:60:59:bf sweh@test1.spuddy.org
The key's randomart image is:
+--[ RSA 2048]----+
|Bo*.+ .          |
|+@ +.+ .         |
|* o  .. .        |
|..    ..         |
|...  E  S        |
|...              |
|...              |
|..               |
|                 |
+-----------------+
$ ls -l
total 8
-rw------- 1 sweh sweh 1766 Oct 30 09:30 server_ca
-rw-r--r-- 1 sweh sweh  403 Oct 30 09:30 server_ca.pub

Creating and using signed host keys

Now we can start using this key to sign our host’s key. This host is test1.spuddy.org. We use ssh-keygen (which has grown massively from its origins!) with the -h flag to indicate “host key”. Because it’s a certificate it can have an expiration date associated with it (-V) flag; in this case 52 weeks.

$ sudo ssh-keygen -s server_ca -I key_for_test1 -h -n test1.spuddy.org -V +52w /etc/ssh/ssh_host_rsa_key.pub
Enter passphrase: 
Signed host key /etc/ssh/ssh_host_rsa_key-cert.pub: id "key_for_test1" serial 0 for test1.spuddy.org valid from 2016-10-30T09:43:00 to 2017-10-29T09:44:13
$ ls -l /etc/ssh/ssh_host_rsa_key-cert*    
-rw-r--r-- 1 root root 1332 Oct 30 09:44 /etc/ssh/ssh_host_rsa_key-cert.pub

We now need to tell the server to offer this certificate. In your sshd_config file you need the line

HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub

Now the client also needs to be told about the CA. It does this my adding the CA’s public key into the known hosts file. This can be the clients’s global entry:

$ cat /etc/ssh/ssh_known_hosts
@cert-authority *.spuddy.org ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/jf9ZrH1qtVB6Gxo31bPA77WGuJ92jb9EaWNp+ht9j+s5+ZLLUm8RM1+T8vBow54IF0uavv/YLcchKIa6vmZdKmMJU5HF6SEdplGqDMDdvd+6U/cm/NQbmS0uTsLQBnCh4zptJJKeC1lc9IOCMwnz1oVA6rGdUc09rZMIDYyW2QpeFArgye5Yi57OWGJl+61QeqBMoAy/hytrYbGXYQsra5c5+PpYdPn2MbXzXY5dLf4H9usyhB+xzx8iAk5Sym4HBL6DnTnOzKAXL5fkI9XKiS3u7Hb1+7CSc0gOYoEFS+8L8kZ5iAqK5kkA0ekQO8DhzPEyZnecqx9n1C6UVb5D sweh@test1.spuddy.org

Now let’s see if this works. I have no known hosts file of my own:

$ cat .ssh/known*
cat: .ssh/known*: No such file or directory
$ ssh test1.spuddy.org
sweh@test1.spuddy.org's password:

Hey, it didn’t ask me to accept the key!

Now I deliberately mis-signed the key for test2. I specified -n badkey.spuddy.org instead of -n test2.spuddy.org.

What happens now?

$ ssh test2.spuddy.org
Certificate invalid: name is not a listed principal
The authenticity of host 'test2.spuddy.org (10.0.0.159)' can't be established.
RSA key fingerprint is 58:45:95:16:c0:98:aa:5f:68:99:36:38:66:18:79:80.
Are you sure you want to continue connecting (yes/no)?

Nice, it warns that the certificate doesn’t match and then falls back to letting you accept the key. Let’s fix the cert and try again:

$ ssh test2.spuddy.org
sweh@test2.spuddy.org's password:

Better!

Client keys

Now how can we use CA keys to login to the server. This is very similar to signing host keys, but without the -h flag, and we specify the username this key is valid for.

First I’ll generate my ssh key and then sign it:

$ cd
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/sweh/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/sweh/.ssh/id_rsa.
Your public key has been saved in /home/sweh/.ssh/id_rsa.pub.
The key fingerprint is:
60:a7:fe:05:7a:7d:d6:ba:fe:2e:e8:05:66:36:fa:61 sweh@test1.spuddy.org
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|      o .        |
|     . +         |
|      . S *      |
|     . . B o .   |
|      o o E.+ .  |
|       o +.=..   |
|        ..o.+=o  |
+-----------------+
$ ssh-keygen -s ssh-ca/server_ca -I user_sweh -n sweh -V +52w .ssh/id_rsa.pub
Enter passphrase:
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh valid from 2016-10-30T10:14:00 to 2017-10-29T10:15:01

We need to configure the server to accept signed keys; the CA’s public key needs to be accessible and sshd_config needs an extra line:

TrustedUserCAKeys /etc/ssh/server_ca.pub

Now let’s try:

$ ssh test2.spuddy.org
Enter passphrase for key '/home/sweh/.ssh/id_rsa': 

test2$ cat .ssh/authorized_keys     
cat: .ssh/authorized_keys: No such file or directory

It let me in!

In /var/log/secure I can see:

Oct 30 10:19:17 test2 sshd[1967]: Accepted publickey for sweh from 10.0.0.158 port 38092 ssh2: RSA-CERT ID user_sweh (serial 0) CA RSA 04:f2:60:0a:89:c0:66:2d:1c:a9:da:ca:11:60:59:bf

This also works with ssh-agent

$ eval `ssh-agent`
Agent pid 2286
$ ssh-add
Enter passphrase for /home/sweh/.ssh/id_rsa: 
Identity added: /home/sweh/.ssh/id_rsa (/home/sweh/.ssh/id_rsa)
Certificate added: /home/sweh/.ssh/id_rsa-cert.pub (user_sweh)
$ ssh test2.spuddy.org echo hello
hello

Summary

Signed host keys can help in avoiding need to maintain a track of host keys, and so can solve the “initial trust” issue problem. It can also solve the “host rebuild” issue. However an automation system to sign these keys when a server is built is essential for this to work.

Similarly we don’t need to distribute authorized_keys files if the user can present a signed certificate. This requires a request system (preferably self-service) where the user can present their public key and get a signed result.

Certificate authentication isn’t supported by all clients; you can expect it in OpenSSH 6.2 onwards, and companies such as RedHat may have backported it to older versions. However desktop based ssh clients such as putty may not support it.

Certificate authentication isn’t as flexible as standard key based authentication in that we don’t have an authorized_keys entry to add restrictions (from=, command=, etc) but it can solve a number of problems, particularly around host key management and distributing of authorized_keys files for human accounts; build a new server, get the host key signed and automatically we have a trust and an authentication process working.

We just need the workflows to do the signing :-)

Update

Based on blog posts and emails (thank you!) there’s a follow up post which goes into some more detail on the advanced use of certificates, including multiple principals and additional restrictions.

Security Headers on HTTP requests

Sun, 23 Oct 2016 11:45:43 -0400

Modern web browsers have a number of settings to help protect your site from attack. You turn them on by use of various header lines in your HTTP responses. Now when I first read about them I thought they were not useful; a basic rule for client-server computing is “don’t trust the client”. An attacker can bypass any rules you try to enforce client side.

But then I read what they do and realised that they are primary to help protect the client and, as a consequence, protect your site from being hijacked. For example, if you can define exactly where javascript can be run from then an attacker will find it harder to inject rogue scripts and steal credentials. We can help stop XSS attacks.

What these headers do is stop the legitimate normal user of your site being used as an attack vector. This makes them an import part of the controls surface of your web presence. You still need to make sure your app isn’t vulnerable to SQL injection, buffer overflow and everything else; these headers (mostly) protect against a third-party attack.

What header values are available?

Fortunately, Scott Helme has done an excellent series of posts that describe the headers. I won’t repeat all of his work, but I will summarise and link to the relevant pages. They’re well worth reading.

Server

This one is simple to understand; if you report the version of the software you’re using then attackers can easily determine what vulnerabilities may be present. Knowing you run Apache 1.2.3 gives a lot more information to attackers then if you reported “FooBarBaz”.

Strict-Transport-Security

This tells the client to always use https for your site. It can cache this value so even if the user types in a non-https address the browser will automatically switch to the TLS version.

X-Content-Type-Options

This one really shouldn’t be needed, but is due to Microsoft being too clever. When you send a file via a https connection you include a content-type header. This should be used to help the browser decide what to do with it. But Internet Explorer ignores this and looks at the data itself; if it looks like a word document then it’ll open it in word, even if the content-type says differently. This header allows you to turn off that feature.

X-XSS-Protection

Modern web browsers have a level of built in protection to stop cross-site scripting. This header turns it on.

X-Frame-Options

When I first came across this it annoyed me. It stopped me from using my own ‘bookmarks’ frame. For the past 20 years my home page was a framed site; on the left were my normal bookmarks and on the right was where the content loaded. Very very old school. X-Frame-Options stopped this working on sites like Dilbert or GPF. Grrr. But it has a good rationale; it stops a site from being embedded in a frame and so can protect against click-jacking. Grump.

Public-Key-Pins

This allows you to restrict what SSL certificates can claim to be you. For example, if you know you will always get a cert from Lets Encrypt then you can flag this. In this way you can help protect against a rogue CA also releasing a cert under your name. We’ve seen this happen before.

Content-Security-Policy

This is a biggie and important protection and is easily misconfigured and can break your site. And, indeed, first time I hit Scott’s site it wasn’t working properly because site changes weren’t being properly reflected in the policy.

It can also stop you from doing stuff; e.g. I can’t embed YouTube videos on this site because the policy I wrote locks it down very tightly.

You will spend most of your time on this entry and will have to do the most testing.

My configuration

Because I use Apache I can’t turn off the whole Server header, but I can remove the version number. But I can do pretty well for the rest:

ServerSignature Off

Header set Strict-Transport-Security "max-age=31536000; includeSubdomains"
Header set X-Content-Type-Options "nosniff"
Header set X-XSS-Protection "1; mode=block"
Header set X-Frame-Options "SAMEORIGIN"
Header set Content-Security-Policy "default-src 'self' 'unsafe-inline' 'unsafe-eval' disqus.com *.disqus.com; style-src 'self' *.googleapis.com *.gstatic.com 'unsafe-inline'; font-src 'self' *.gstatic.com"

Testing

Scott has also created a site for you to be able to test your setup. Just go to https://securityheaders.io, enter your site name and get an almost immediate response.

I don’t have my cert pinned at the moment. You can also note that I have a warning for the CSP; I allow ‘unsafe-inline’. So my configuration isn’t necessarily safe, but because I’m forcing SSL then the chances of an attacker injecting rogue scripts isn’t really any different to trusting self and having the code loaded from an external page.

Summary

Most of these headers can be turned on without any impact on your app; forcing TLS, turning off content-type sniffing, turning on XSS protections and frame clickjacking protectings are mostly safe (not 100% because a site may trip things and need workarounds). CSP is the hard one, but definitely worth spending time on.

Scoring an A+ for SSL/TLS

Sun, 16 Oct 2016 16:02:28 -0400

(Side note: in this post I’m going to use TLS and SSL interchangably. To all intents and purposes you can think of TLS as the successor to SSL; most libraries do both).

You can think of security as a stack. Each layer of the stack needs to be secure in order for the whole thing to be secure. Or, alternatively, you can think of it as a chain; the whole thing is only as strong as the weakest link.

No matter how you look at it, you need to ensure that every part of the environment is good.

In previous posts I’ve talked about static code analysis, and system scanning. This can help make sure your program hasn’t any obvious gaps, and that you’re not running a bad alternate path.

Now let’s look at protecting the data between you and your users. Typically we do this with SSL. So you get your certificate and click the “turn on SSL” option and you’re done; right?

Unfortunately not. SSL has a long and not so clean history. There have been both implementation bugs and logic issues. “Heartbleed”, “BEAST”, “POODLE”… at one point it seemed that an issue was found in SSL every other week.

Further, something that was considered strong 10 years ago isn’t strong today (RC4 codes). And new and better methods of protecting data now exist (elliptical curve, forward secrecy).

So we may need to tune your SSL setup as well. Turn off weak settings, add new configurations.

The problem is in how to know if you’ve done a good job?

Testing

Fortunately there’s a very useful web site run by Qualys: https://www.ssllabs.com/ssltest/

This will run through your site, testing for common failure modes. It’ll try common bugs, weak settings and even provide a good summary of how well your site will work with common browsers. It takes somewhere around 90-120 seconds to run.

My site (PDF) scores an A+ on these tests and it flags some compatibility issues.

e.g. “This site works only in browsers with SNI support.” That’s because I present a different SSL certificate for each virtual host, and for this to work we need the SSL “Server Name Indication” extention. Pretty much every common browser supports this these days.

If we go down further into the report it tells me the Android 2.3.7, IE6, IE8, Java 6 don’t support SNI. I can live with that!

Further it tells me that IE6 and IE8 just won’t work with my site because of other settings. Do I care? According to a browser market share report IE8 is under 4% of usage. I can live with that.

My settings

I use Apache with OpenSSL, which allows you to specify the protocols and cipher suites. The first is to ensure SSLv2 and SSLv3 are turned off. Then we can pick the ciphers we want:

SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder on

SSLCipherSuite ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:HIGH:!EDH-RSA-DES-CBC3-SHA:!DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4

Conclusion

These settings aren’t necessarily the best. I could tighten it up if I don’t care about compatibility with older tools (java6 and java7 might be easy to discard).

These settings are also not always going to be fixed; a few months back I did not have !EDH-RSA-DES-CBC3-SHA:!DES-CBC3-SHA in the Cipher suite because these used to be considered good. It’s worth reviewing the results of the SSLlabs test on a regular basis. And definitely after a new vulnerability has been exposed.

Getting an A or an A+ is easy, and is an essential part of the complete security solution for your web site. There are many good reasons to do it!

Deep scanning your deployment

Sun, 09 Oct 2016 09:02:20 -0400

In my previous post I wrote about some automation of static and dynamic scanning as part of the software delivery pipeline.

However nothing stays the same; we find new vulnerabilities or configurations are broken or stuff previously considered secure is now weak (64bit ciphers can be broken, for example).

So as well as doing your scans during the development cycle we also need to do repeated scans of deployed infrastructure; espcially if it’s externally facing (but internal facing services may still be at risk from the tens of thousands of desktops in your organisation).

In this spirit I decided to scan this server using OpenVAS and that was installed as part of the the Kali Linux build. OpenVAS can also call other tools (eg Nikto, that I mentioned previously).

Now I’d done this back in May and got a single “low level” message about TCP timestamps, so I didn’t expect to see anything new.

So let’s do a run:

Well, damn! That’s not right! I get an A+ on ssllabs tests (more on that in a later post). What’s going on?

Oh… port 563? That’s NNTPS. Oh! I’d recently installed an SSL news server on this machine! And it seems the defaults aren’t so secure.

So I dig into the manual page to work out how to disable SSLv2, SSLv3 and also to set the cipher suite to match that of my apache server.

At the same time I told OpenVAS to update its libraries of known threats and issues. A scanner with an stale database isn’t really much use, and OpenVAS has an update mechanism, with advisories being fed from DFN-CERT. This is pretty impressive, all the more so that it’s free.

So, problem solved, right?

Damnit! Now my web server is also being flagged! And a new DH issue. Interestingly the TCP Timestamp issue has gone away; maybe that was no longer really considered a problem. Evaluations change.

So what is weak…

Now this is annoying. The values reported here don’t match the config entries used by OpenSSL.

Are these weak? The nmap -p 443 hostname --script +ssl-enum-ciphers routine reports those entries in the output and considers them strong. As we’ve seen, though; evaluations can change. This is why a scanning tool needs update abilities. I’m more likely to believe the OpenVAS “weak” report as being newer and more accurate than the builtin nmap rules that RedHat won’t have updated in forever.

Now my SSL configuration included entries that mentioned DES:

ECDHE-RSA-DES-CBC3-SHA
EDH-RSA-DES-CBC3-SHA
DES-CBC3-SHA
!DES

This was to allow IE6 to talk to my site. But let’s remove the three entries and run again. Same result! Still an error.

Some more digging and it seems that the bad entries were brought in with the HIGH setting. So now we needed

HIGH
!EDH-RSA-DES-CBC3-SHA
!DES-CBC3-SHA

The ECDHE entry I’d previously removed wasn’t at fault. That’s elliptical curve. If only the output of the “weak ciphers” test and the configuration options matched!

Woohoo! Now all the cipher problems have been solved.

Umm, there’s a downside to this. IE6 and IE8 can no longer reach my site. Do I care? According to a browser market share report IE8 is under 4% of usage. I can live with that.

But what about this DH issue? I looked into the server code and it seems to respond with different key sizes depending on the block size. I’m not sure the code is correct, but I’m not an SSL expert here. However this is a “low value” service; the data transported is passed in clear text on port 119 and the username/passwords used to access this server may also be used in plain text on 119. So I took the decision to “accept the risk” and entered an override for my servers for this issue on this port, reducing the severity to “Log”

The final result is a score of zero! Yay!

Notice those 36 log entries. Now some of these may be worth looking at.

Here’s our override, as a log entry. There’s also an entry claiming HSTS isn’t present. Which is a lie! So let’s look at this:

Ah! This is a response to a ‘bad request’ and it doesn’t see the HSTS header. But a real request does:

$ wget -O /dev/null --server-response https://www.sweharris.org/ 2>&1 | grep Stri
  Strict-Transport-Security: max-age=31536000; includeSubdomains

So we can see this is a false positive; I get an A rating for header security at securityheaders.io.

Lessons Learned

Keep scanning servers deployed to production; even if the app doesn’t change something else might
Keep your scanning tool databases up to date!
Different tools may report different evaluations for the same issue
Configuration parameters don’t match reported items
Sometimes you have to accept losing customers (IE8?) to be secure
Sometimes you can just accept the insecurity risk. That’s why we have “risk management” teams!

OpenVAS is a noisy scan (it flooded my Apache logs with probe attempts, caused multiple ssh failures, triggered firewall protection rules). In an enterprise environment you might want to allow-list the traffic from your scanning servers so they don’t trigger SOC alerts and don’t flood out real alerting traffic. But despite this a full vulnerability scan is useful.

It definitely alerted me to a weakness on my server that I wasn’t aware of!

Scanning your code

Sun, 02 Oct 2016 19:53:09 -0400

In many organisations an automated scan of an application is done before it’s allowed to “go live”, especially if the app is external facing.

There are typically two types of scan:

Static Scan
Dynamic Scan

Static scan

A static scan is commonly a source code scan. It will analyse code for many common failure modes. If you’re writing C code then it’ll flag on common buffer overflow patterns. If you’re writing Java with database connectors it’ll flag on common password exposure patterns. This is a first line of defense. If you can do this quickly enough then it might even be part of your code commit path; eg a git merge must pass static scanning before it can be accepted, and can be a triggered event.

Static scan’s aren’t perfect, of course. Nothing is! It can miss some cases, and can false-positive on other cases (requiring work-arounds). So a static scan isn’t a replacement for a code review (which can also look at the code logic to see if it does the right thing), but is complementary.

Static analysis was one reason for a massive increase in security of opensource projects by pro-actively flagging potential risks.

Dynamic scan

A dynamic scan actually tests the component while it’s running. For a web site (the most common form of publically exposed application) it can automatically discover execution paths and fuzz input fields and try to present bad data. Dynamic scanning can only test the code paths it knows about (or discovers). Stuff hidden behind authentication must be special cased. And it could potentially caused bad behaviour to happen; it could trigger a little Bobby Tables event, so be very careful about scanning production systems!

Common scanning

A common practice is to combine both types of scan. A static scan is typically quick and can be done frequently. A dynamic scan may take hours, and so may not be done every time.

So a common workflow may be:

Developer codes, tests stuff out in a local instance
Developer commits Potential static scan here
“Pull request”
Code review
Code merge Static scan here

So far work has been done purely in development; no production code has been pushed. The developer is allowed to develop and test their code with minimal interruption. It’s only when code is “ready” that scanning occurs.

Now depending on your environment and production controls things may get more complication. Let’s assume a “merge” then triggers a “QA” or “UAT” build.

Merge fires off “Jenkins” process
Jenkins stands up a test environment
Static scan occurs
Merge successful only if the scan is successful.

Depending on the size of complexity of the app this scan could take hours. One advantage to micro-services is that each app has a small footprint and so the scan should be a lot quicker.

It’s important for a developer to not try and hide code paths. “Oh this scanning tool stops me from working, so I’ll hide stuff so it never sees it”… We all have the desire to do things quicker and know we’re smarter than “the system”, but going down this path will lead to bugs.

Side bar: There’s the standard open-source comment; “with sufficient eyes, all bugs are shallow”. There’s a corollary to this; “with sufficient people using your program all bugs will be exploited”.

Scanning this site and false positives

Now this site is pretty simple. I use Hugo to generate static web pages. That means there’s no CGI to be exploited; no database connections to be abuse. It’s plain and simple.

But does that mean a scan of the site would be clean? Is the Apache server configured correctly? Have I added some bad CGI to the static area of the site? Have I made another mistake somewhere?

The Kali Linux distribution includes a tool called Nikto (no bonus points for knowing where the name comes from). This is a web scanner that can check for a few thousand known issues on a web server. It’s not a full host scan (doesn’t check for other open ports, for example).

WARNING: Nikto is not a stealth tool; if you use this against someone’s site then they WILL know.

Given how my site is built, I wouldn’t expect it to find any issues.

Let’s see!

+ Server: Apache
+ Server leaks inodes via ETags, header found with file /, fields: 0x44af 0x533833f94c500 
+ Multiple index files found: /index.xml, /index.html
+ The Content-Encoding header is set to "deflate" this may mean that the server is vulnerable to the BREACH attack.
+ Allowed HTTP Methods: POST, OPTIONS, GET, HEAD 
+ OSVDB-3092: /sitemap.xml: This gives a nice listing of the site content.
+ OSVDB-3268: /icons/: Directory indexing found.
+ OSVDB-3233: /icons/README: Apache default file found.
+ 8328 requests: 0 error(s) and 7 item(s) reported on remote host

Interesting, it found 7 items to flag on.

Except if we look closer, these are not necessarily issues. Is the “inode number” something we need to care about? I don’t think so. The sitemap.xml file is deliberately there to allow search engines to find stuff. The /icons directory is the apache standard… is there a risk to having this exposed?

So even a “clean” site may have issues according to the scanning tool, even though it really doesn’t. (At least I assume it doesn’t! My evaluation may be wrong…)

It takes a human to filter out these false positives. If you’re going to put this as part of your CI/CD pipeline then make sure you’ve filtered out the noise first!

Summary

Static and Dynamic scanning tools are complementary. They both have strengths and weaknesses. Both should be used. Even if your code is never exposed to the internet it’s worth doing it, to help protect from the insider threat and also to mitigate against an attacker with a foothold from being able to exploit your server.

They do not replace humans (e.g. code review processes) but can help protect against common failure modes.

ADDITIONAL

2016/10/13 The ever interesting Wolf Goerlich, as part of his “Stuck in traffic” vlog, also points out another limitation of scanning tools; they don’t handle malicious insiders writing bad code (and, by extension, and intruder able to get into the source code repo and make updates). You still need humans in the loop (e.g. code review processes), whether it’s to test for code logic or malicious activities. You should watch Wolf’s vlog.

Kerberos keytab management

Sat, 24 Sep 2016 18:36:41 -0400

In an earlier blog I wrote that SSH keys need to be managed. It’s important!

In the comments I was asked about keytabs. I felt this such a good question that it deserved it’s own blog entry.

The basics of Kerberos auth

In essence, kerberos is a “ticket based” authentication scheme. You start by requesting a TGT (“Ticket granting ticket”). This, typically, is where you enter your password. This TGT is then used to request service tickets to access resources. There’s a description which goes into a lot more detail, but is pretty readable.

Now all of this is pretty invisible and runs under the covers; the user never needs to see any of this. Indeed, Windows Active Directory is kerberos based!

So let’s see this in action:

$ ssh sweh@kclient
sweh@kclient's password: 

kclient$

So far that looks just the same as any other login! But if we look deeper…

kclient$ klist
Ticket cache: FILE:/tmp/krb5cc_500_WfoXRC1791
Default principal: sweh@SPUDDY.ORG

Valid starting     Expires            Service principal
09/24/16 18:46:08  09/25/16 18:46:08  krbtgt/SPUDDY.ORG@SPUDDY.ORG
        renew until 09/24/16 18:46:08

I have a TGT associated with my login. That was all transparent. And I can use this to login to other machines…

kclient$ ssh kdc
kdc$ exit
Connection to kdc closed.
kclient$ klist
Ticket cache: FILE:/tmp/krb5cc_500_WfoXRC1791
Default principal: sweh@SPUDDY.ORG

Valid starting     Expires            Service principal
09/24/16 18:46:08  09/25/16 18:46:08  krbtgt/SPUDDY.ORG@SPUDDY.ORG
        renew until 09/24/16 18:46:08
09/24/16 18:47:56  09/25/16 18:46:08  host/kdc.spuddy.org@SPUDDY.ORG
        renew until 09/24/16 18:46:08

Well, it didn’t ask me for my password… and now I can see there’s a service ticket for that host.

So Kerberos can give us a form of “single sign on” (I only had to enter my password the once), and it’s all nice and transparent.

What is a keytab?

Everything so far has depended on being able to enter my password to get that initial TGT. This is fine for humans, but for automated processes (e.g. oracle wanting to scp files between servers) then it’s not very useful.

So we have the ability to use a keytab file to get that initial key.

In my test environment I created a user1 and extracted that user’s key into a keytab file:

$ kadmin -p sweh/admin
Authenticating as principal sweh/admin with password.
Password for sweh/admin@SPUDDY.ORG: 
kadmin:  ktadd -k user1.keytab user1@SPUDDY.ORG
...lines deleted...
kadmin: exit
$ ls -l user1.keytab 
-rw------- 1 sweh sweh 666 Sep 24 18:53 user1.keytab

Now we can use this to login to kerberos:

$ kinit -k -t user1.keytab user1@SPUDDY.ORG
$ klist
Ticket cache: FILE:/tmp/krb5cc_500_WfoXRC1791
Default principal: user1@SPUDDY.ORG

Valid starting     Expires            Service principal
09/24/16 18:55:29  09/25/16 18:55:29  krbtgt/SPUDDY.ORG@SPUDDY.ORG
        renew until 09/24/16 18:55:29

We can see that we didn’t get asked for a password, and the “Default principal” line mentions user1. As far as the kerberos infrastructure is concerned, I now am user1 and can use this to ticket to login as him:

$ ssh user1@kclient
Last login: Sat Sep 24 18:56:39 2016 from kclient.spuddy.org
[user1@kclient ~]$ exit
logout
Connection to kclient closed.
$ klist
Ticket cache: FILE:/tmp/krb5cc_500_WfoXRC1791
Default principal: user1@SPUDDY.ORG

Valid starting     Expires            Service principal
09/24/16 18:55:29  09/25/16 18:55:29  krbtgt/SPUDDY.ORG@SPUDDY.ORG
        renew until 09/24/16 18:55:29
09/24/16 18:56:39  09/25/16 18:55:29  host/kclient.spuddy.org@SPUDDY.ORG
        renew until 09/24/16 18:55:29

So we can see that these keytab files are as powerful as a passphrase-less ssh private key.

In some respects they are more powerful than ssh keys, because there’s no server side restrictions that can be applied; with ssh public key authentication the public key needs to be placed on the host for it to be usable, and we can limit where the private key can be used (“from=”) and what it can do (“command=”). With kerberos authentication we can login to any server where this ticket is valid (pretty much anywhere the account exists) and get a shell.

So it becomes critically important that keytab entries of this type are closely managed. Maybe they should be put into an equivalent of a password vault and only pulled “as needed” by the application? Or, at least, placed into a locked directory where no one except the app can reach it.

This is one reason why NIST recommends SSH keys over kerberos in many cases.

Service keytabs

There’s a second form of keytab file; these are service entries and are typically of the form service/FQDN@domain. Your kerberos host probably already has one in /etc/krb5.keytab for the host service

-rw------- 1 root root 448 Sep 24 18:21 /etc/krb5.keytab

You may have noticed the host/kclient.spuddy.org entries in the klist entries above… that’s as a result of this type of service.

If you want to do kerberos authentication for web sites you would create a HTTP service principal and then that keytab file would be used by the web server to verify incoming requests.

Now these keytab entries are used to verify the service you’re talking to is the service you expect to talk to. In a way they are comparable to the ssh host keys or SSL certificate, but each service has their own key. Because of this identity assertion these keytab files should also be carefully managed.

k5login and k5users files

There’s one final “gotcha” with kerberos; it allows one user to trust another. In $HOME/.k5login you can list other kerberos principals that are allowed to login to this account.

So user2 has the following entry:

$ sudo cat ~user2/.k5login
user1@SPUDDY.ORG

Now this means that if I have a user1 TGT I can login as user2! We’ll re-use the keytab file from earlier…

$ kinit -k -t user1.keytab user1@SPUDDY.ORG
$ ssh user2@kclient
Last login: Sat Sep 24 18:25:45 2016 from kclient.spuddy.org
[user2@kclient ~]$

In this respect the .k5login file is close to the ssh authorized_keys file; if I could add my login to ~oracle/.k5login then I can login as oracle without needing a password.

And, of course, we have NFS weaknesses if the user home directory is on an NFS server…

Worse, this file also works with ksu…

$ kinit -k -t user1.keytab user1@SPUDDY.ORG
$ ksu root
Authenticated user1@SPUDDY.ORG
Account root: authorization for user1@SPUDDY.ORG successful
Changing uid to root (0)
[root@kclient sweh]# cat /root/.k5login 
user1@SPUDDY.ORG

Also note that ksu will also work with .k5users, which is similar but allows for restrictions on the command. Do a man ksu for details.

So just as how with ssh you need to set the AuthorizedKeysFile file entry, in /etc/krb5.conf you should set k5login_directory directory to something users can not modify themselves, and then have a management tool to control them, just as you would control ssh public keys.

I’m not sure this is fully supported, though. I’ve had marginal results (ssh works properly, ksu doesn’t?). So you may want to verify this change actually works the way you want and if it doesn’t then create some form of detective control to see if a .k5login or .k5users file has been created.

Summary

Unless you have a specific need for kerberos (eg a kerberised file system where you need a ticket to access data), I’m not a fan.

We can see that for “service accounts” we have a keytab management problem to be resolved.

For human accounts there’s a tendancy to use Active Directory as the source of kerberos tickets, but I’ve previously written why I think this is a bad idea.

But if you do want to go down the kerberos path (and it may be reasonable for application servers, such as web servers) then we need to manage keytabs.

If you’re looking at a key management solution (e.g. one that manages SSL certs) then it’s worth seeing if it can also manage SSH keys and kerberos keytabs. It’s almost the same problem…

SSH Password exposure

Sun, 18 Sep 2016 11:53:14 -0400

Over on the Unix/Linux Stackexchange someone asked if ssh’ing to the wrong machine could cause their password to be exposed.

I answered “yes”… and was surprised to see the answer get over 140 upvotes.

Demonstration

So it seems as if there’s some interest in this. I decided to demonstrate this risk in a short practical session.

Let’s start with a generic CentOS 7 install. It’s got nothing special on it.

We want to replace sshd with a modified one that has a one line change:

*** openssh-7.3p1.orig/auth-passwd.c    Wed Jul 27 18:54:27 2016
--- openssh-7.3p1/auth-passwd.c Sat Sep 17 19:34:41 2016
***************
*** 84,89 ****
--- 84,90 ----
  auth_password(Authctxt *authctxt, const char *password)
  {
        struct passwd * pw = authctxt->pw;
+       logit("User %s Password %s",pw->pw_name,password);
        int result, ok = authctxt->valid;
  #if defined(USE_SHADOW) && defined(HAS_SHADOW_EXPIRE)
        static int expire_checked = 0;

We build this and copy the sshd over the top of the original.

So let’s try and login to this server:


    % ssh -l fred test1
    fred@test1's password: 
    Permission denied, please try again.
    fred@test1's password: 
    Permission denied, please try again.
    fred@test1's password: 
    Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

Now on the server, we can see lines in /var/log/secure:

    Sep 17 19:36:33 test1 sshd[3597]: User fred Password test
    Sep 17 19:36:34 test1 sshd[3597]: User fred Password test2
    Sep 17 19:36:36 test1 sshd[3597]: User fred Password test3

They are the three passwords I entered!

Now if the user doesn’t exist on the server we get NOUSER as the username, which isn’t helpful. Fortunately other lines retain the data:

    Sep 17 19:40:38 test1 sshd[3601]: User NOUSER Password hello
    Sep 17 19:40:38 test1 sshd[3601]: Failed password for invalid user fred2 from 10.0.0.137 port 42746 ssh2
    Sep 17 19:40:39 test1 sshd[3601]: User NOUSER Password there
    Sep 17 19:40:39 test1 sshd[3601]: Failed password for invalid user fred2 from 10.0.0.137 port 42746 ssh2
    Sep 17 19:40:40 test1 sshd[3601]: User NOUSER Password eevryone
    Sep 17 19:40:40 test1 sshd[3601]: Failed password for invalid user fred2 from 10.0.0.137 port 42746 ssh2

So there’s still sufficient data to pick up all the usernames and passwords being attempted. And the connecting user is none the wiser that their password might have just been stolen

Is this paranoia?

Everything you do on a server is inherently dependent on trust on the SA on that server. They could keyboard sniff, or look in your process address space or impersonate you or…

So if we have an inherent trust on the SA, why is this whole “steal password via ssh” question a big deal?

Firstly, an attempt to login to the wrong machine can expose your credentials. Now it’s not an SA on your machine you’re shown your password to, it’s some other admin who you may never have heard of.

Secondly, credentials get re-used; especially in a large enterprise where there may be a centralised LDAP server or other service. Federation or even direct integration between systems (Unix talking to Active Directory) could let you cross environments. A single typo in a hostname could leak your password to many other systems an attacker.

Mitigation

The primary mitigation is to never use passwords. The problem with password based authentication is that the remote server has to have the plain text available so that it can encrypt it according to the values in /etc/shadow, or for PAM based authentication, or… this means the server process is in an inherent position of trust.

If you use ssh public key authentication, or kerberos authentication then your password is never passed to the server. The attacker will never see it.

You can force this in your .ssh/config file by ensuring that only the preferred authentication mechanisms are permitted

Host *
  PreferredAuthentications publickey

You probably want to take other precautions even if you are logging into the right server; e.g. disable agent forwarding so the remote SA can’t abuse your agent; e.g. don’t request kerberos ticket forwarding so there’s no kerberos ticket on the server for the remote SA to abuse.

Secondary mitigation is to be very careful about the known_hosts values. If you suddenly get prompted to accept a key then it’s likely you’re talking to a server you’ve never spoken to before. BEWARE! Of course this doesn’t help if it’s the first time you’re logging in…

You can create ‘shortcuts’ for yourself; eg in .ssh/config you can create entries such as

Host myhost
  Server some.long.domain.name

Now ssh myhost will go to the right place, and be less likely to typo.

Conclusions

“Passwords bad, m’kay?”.

We’ve known for a long time that passwords are a terrible means of authentication. Mostly we’ve been saying that because they’re something that humans have to remember, and this makes them brute-forcable. Add in the bad practices, to which NIST have finally said “stop doing that!”, and we have a headache.

But on top of this, we can see that passwords are bad from a purely technically perspective.

The problem with public key authentication is that the keys need to be properly managed. I cover this in more detail but the essence is that you need a trusted “out of band” method of ensuring your public keys are distributed to servers you’re allowed to access, and management of host keys becomes complicated (especially in a dynamic re-build environment).

For my own machines I only use ssh public keys; my “build” process copies my keys so they’re already present. I never expose my password like that. But while this may work for individuals it definitely doesn’t scale for SMEs, let alone larger companies. Imagine doing that for 50,000 servers and 10,000 people.

So we need a management solution for these keys. It’s the only way we can scale, be dynamic, and stop passwords from being used.

HSMs, what are they good for?

Tue, 13 Sep 2016 08:51:07 -0400

What is a HSM?

A HSM is a hardware device that can perform cryptographic functions in a “secure” manner.

The idea is that you can load your private key into a HSM and be sure that it’s safe from theft. Anyone tries to physical access the device and it’ll wipe (or literally burn) the data. So encryption or signing of data can be trusted because only the HSM has the key needed to do the crypto.

So, for example, there are modules for Oracle to allow for transparent data encryption to happen via a HSM. There are certificate authorities that have their root key in a HSM. International money transfer or trading may be signed and verified by HSMs.

Basically, HSMs are a big thing.

The problem

But the problem I have isn’t with the HSM, itself, but in how they are accessed.

Take a standard network based HSM. There are two standard ways a client machine can access the HSM:

IP based. The server is “trusted” by the HSM to make requests. (an on-machine HSM card is really just a variation of this; we use the local hardware bus rather than the network).
Authentication based. The server authenticates to the HSM (eg with an X509 cert) in order to make requests. There may be variations of #2 (eg symmetric encryption) but it’s all “private data” based.

This second method is becoming more common with dynamic resources. Think “cloud”; you might spin up 10 servers and get assigned IP addresses, then shut down 3 of them, then spin up 5 more with different addresses. At this point you can’t use IP address as a basis of trust, so we use an authentication protocol instead.

Attack paths

An IP based trust to the HSM is naturally exposed to any system administrator on the machine. Or any attacker that manages to get remote code executition. They can directly call the HSM since they’re coming from a forged source.

An authentication based approach runs the risk of the credential being stolen. Either via an exploit in the app or OS, or via whatever mechanism is used to provide credentials to the app. In this scenario the attacker may be able to make calls to the HSM from any machine on the trusted network.

In both of these scenarios the attacker can now call the HSM and have it perform the necessary calculations using the stolen access.

Where HSMs are useful

I can think of two ways that HSMs are useful… and these come into play once you have already been attacked.

So, assume your attacker has got an RCE to a trusted machine or has managed to steal an authentication credential.

At this point you can revoke access to the HSM from the bad machine, or revoke the bad credential. Now the attacker no longer has access to the HSM and thus to the private key. They can no longer forge requests or decrypt data.

Without a HSM the attacker may be able to steal the primary keys and so continue bad activity offline (forge requests, decrypt exfiltrated encrypted data), but with a HSM their access to the keys have been revoked. Sure, this stops your app from running… but you probably want that, anyway!

This makes it easier to limit the scope of intrusion and to recover from it (fix the vulnerability in the code, build a new machine, create new trusts, create new credentials), rather than having to revoke the primary encryption keys, re-encrypt any data, ensure any counter-parties are aware the key is no longer valid…

Sure, there’s still clean up and rollback type activities to be done after an intrusion but at least the key, itself, is still secure.

Similarly, HSMs help by limiting attack paths to machines that can see the HSM; if there’s no IP route between an attacker and a HSM then they can’t make requests. The attacker may have been able to exfiltrate the authentication secret to talk to the HSM, but since they can’t reach the HSM from outside the network then they can’t make use of it. They need to maintain a persistent intrusion path, which may make it easier to be detected.

How does the cloud change this

Dynamic elastic compute pretty much rules out IP based authentication. In addition some PaaS systems or container ecosystems may use NAT rules so that multiple applications all have the same externally visible IP address.

So we have to use an authentication based approach. This suffers the previously discussed problem of getting the credentials into the application.

You need to make sure that you’re building your virtual networking constructs so as to ensure that only approved IP ranges can reach the HSM (e.g. in a 3 tier architecture, the application and database tiers may be able to reach the HSM, but the web front end tier can’t).

This, of course, raises the question on how well you trust the cloud provider to maintain network security at their end; whether their stuff could abuse hypervisor and control plane access… but this applies to the whole cloud environment!

What about cloud HSMs

Amazon can provision you a Cloud HSM which you can use instead of your own on-premise HSMs. The idea is that they are inside your VPC and so the latency is a lot less and so applications that depend on them can see increased performance.

The Amazon solution is based on SafeNet Luna SA devices, which provide a strong separation between “administrative access” and “application access”. Amazon staff shouldn’t be able to access your secrets in the HSM.

Summary

A HSM doesn’t appear to provide any real improvement for an active attack via an RCE or similar; the attacker can use the credentials and access rights of the application to get the HSM to perform activities.

However the HSM can help limit the consequences by making it easier to revoke and restrict access to the secret key.

The weakest part of a HSM is in the secret management; how do you help prevent leakage of your credentials, while still allowing for automated deployment. In a PaaS environment you may have inherent trust on the PaaS administrators, since they may be providing that initial “identity”.

Question

Are there any other ways a HSM can make things better? Please let me know!

SSH key management

Mon, 05 Sep 2016 10:39:14 -0400

SSH is probably the most common method of logging into Unix machines over a network connection. It has pretty much replaced the older telnet and rlogin protocols because it encrypts data over the network and has other “goodness”.

However one of the “good” parts of ssh is probably the least well managed; ssh keys.

What is an ssh key?

When attempting to login to a server, ssh will attempt multiple different authenticators. Now we all know (and hate) the standard password mechanism.

$ ssh test2
sweh@test2's password: 
Last login: Sun Sep  4 12:02:13 2016 from test1

A few years ago I described some tests on using Kerberos as an authenticator.

I even use Google Authenticator as a form of two factor authentication.

But the big one, and one of the most powerful, are ssh keys.

An ssh key is a public/private key pair. A user can generate their own key with ssh-keygen and then copy the public part to the remote server.

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/sweh/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/sweh/.ssh/id_rsa.
Your public key has been saved in /home/sweh/.ssh/id_rsa.pub.
The key fingerprint is:
ad:cc:ec:e6:9b:7f:ba:96:32:1d:83:b7:b4:1d:cf:00 sweh@test1.spuddy.org
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
|         .E      |
|        S...     |
|       +..= o    |
|        =+ B =   |
|       .+.* o o  |
|       o=*++     |
+-----------------+
$ scp ~/.ssh/id_rsa.pub test2:.ssh/authorized_keys
sweh@test2's password:
id_rsa.pub                                    100%  403     0.4KB/s   00:00

We can now use this to authenticate instead:

$ ssh -i ~/.ssh/id_rsa sweh@test2
Enter passphrase for key '/home/sweh/.ssh/id_rsa': 
Last login: Sun Sep  4 12:10:54 2016 from test1

Indeed, if the id_rsa file exists (or one of a few other names) then it will be selected automatically (with some details that don’t matter, here).

$ ssh sweh@test2
Enter passphrase for key '/home/sweh/.ssh/id_rsa': 
Last login: Sun Sep  4 12:14:51 2016 from test1

Why would you do this?

In the “bad old days” when servers were DES based, and so limited to 8 characters, ssh keys could use much longer passphrases. This benefit has been reduced with modern SHA hashing of passwords :-)

But ssh keys still have some advantages. From a user perspective we can run an agent that can remember the unlocked key and present it automatically. This means the user doesn’t have to keep retyping their passwords:

$ eval `ssh-agent`
Agent pid 1904
$ ssh-add ~/.ssh/id_rsa
Enter passphrase for /home/sweh/.ssh/id_rsa: 
Identity added: /home/sweh/.ssh/id_rsa (/home/sweh/.ssh/id_rsa)
$ ssh test2 echo running remotely with no password
running remotely with no password
$ ssh test2
Last login: Sun Sep  4 12:15:01 2016 from test1

$ logout
Connection to test2 closed.
$ ssh test2 echo third login                      
third login

Another, much more powerful configuration, is that restrictions can be placed on the ssh public key. For example, we can say that it can only be used from a specific server and can only be used to run a specific command.

eg on server test2 I modify the authorized_keys file so it starts:

from="10.0.0.158",command="/bin/echo forced command" ssh-rsa AAAAB3Nz....

Now if I try to use the key for a normal login:

$ ssh test2
forced command
Connection to test2 closed.

We can see the command= stuff was forced and I wasn’t able to do anything else. Similarly if I tried to use the key from a different server then it won’t be permitted.

This is extremely powerful for automation purposes; for example a production server could transfer files to DR on an automated basis using an ssh key that has restrictions like this; the key can only do what it is permitted to do, as opposed to having a password based login that can do anything on the remote machine.

So what problems are there with ssh keys?

No enforcement on passphrase complexity

The passphrase set on a key when we create it has no real complexity rules that can be applied. This is a pure client side activity. Indeed it’s possible to create keys without a passphrase (which is good for automation tasks, such as the DR file copy mentioned above) but a potential risk.

No easy distribution

If I want to login to 100 servers then I need to copy that key to all 100 servers. If a new server is built then I need to remember to copy the key there. Some tools such as ssh-copy-id help to do this, but it’s still a mostly manual task.

No expiration

An ssh key is an authenticator and it can be copied between machines. This means that the private key could be copied around and potentially leaked. At this point the the key is exposed to offline attacks against the passphrase (worse, the key could have been generated without a passphrase!).

In an enterprise environment with thousands of users, all it takes is one exposed key to open a potential entry point for an attacker.

We’ve learned that expiring user passwords is bad because it causes people to pick easy to hack passwords.

However ssh keys are strong cryptographic keys, generated by tools, with no human memory requirements. They can theoretically be expired and a new key forced, but there’s nothing “out of the box” that enables this.

Unauthorized granting of access.

An authorized_keys file can have multiple keys permitting access. This is useful, especially for automation tasks; eg key1 can be used from server1 and can copy files to directory1, and key2 can be used from server2 and can copy files to directory2.

But this also allows humans to add their keys to an account. Let’s say you have an oracle account that your database runs under. You don’t allow people to login directly but force them to go via “break glass” process, in order to control and monitor access to this privileged account. Without any strong controls, the first time I login I could add my personl key to oracle’s authorized_keys file. Now I can login as oracle whenever I like, bypassing break-glass.

Similarly, people tend to not share passwords, but if Fred has a login on a server then he could add John’s public key to his authorized_keys file and now John can login to the server as well… without any authorization or oversight. Your central access admin request system has been totally bypassed.

Keys on NFS servers…

A long standing problem with NFS is that it is an effective “trust the client” environment. The client says “user 12345 wants to access file X” and the server has to trust the client when it says you are user 12345. Now this has been changed, to some extent, with NFSv4 and with kerberos but this isn’t as widely deployed as we might wish.

And commonly, user HOME directories are on these NFS servers. So now it becomes possible for one SA to add their key to another user’s authorized_keys file and now login to other servers that they wouldn’t normally be permitted to. All of the previous problems are exposed by having keys stored on NFS.

So why use it?

If ssh keys have all these problems then why use them?

Mostly, as NIST says, ssh keys are extremely powerful; the command and from restrictions allow you to build solutions that are more tightly locked down than other authenticators.

HOST keys

In addition to the ssh keys we’ve been discussing there’s also the concept of a host key. These are a form of ssh key that’s used to identify the host.

First time you login to a machine you’ll be presented with the public key signature of a host, which you can accept. Next time you login the client will verify the key hasn’t changed; you have a level of trust that the server you’re accessing is the server you meant to access:

$ ssh test2
The authenticity of host 'test2 (10.0.0.159)' can't be established.
ECDSA key fingerprint is 32:31:44:a0:9b:61:1b:ff:2f:db:76:ae:a9:a5:36:2b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'test2,10.0.0.159' (ECDSA) to the list of known hosts.
forced command
Connection to test2 closed.

$ ssh test2
forced command
Connection to test2 closed.

This is (mostly) OK, but what happens if a server is rebuilt? Now the key will change. In the modern dynamic world (eg dynamic building/destroying of Amazon EC2 instances) your host keys may change all the time.

Ideally we need a way to manage these host keys so that the client can determine the current host key; this will solve the “trust key presented on first connect” and solve the “key changes on rebuild” problem.

Conclusion

Some of the problems with ssh keys can be mitigated by changing the default config. Instead of having the authorized_keys file in the user’s home directory, the configuration can point to a location that only root can modify.

e.g. /etc/ssh/sshd_config could read

AuthorizedKeysFile /etc/ssh/auth_keys/%u

Each file in that directory should be owned by root:root and have permission 0644 (-rw-r--r--). Now Fred can not grant John access to the system, unauthorized access to oracle is prevented, NFS risks have gone away…

We haven’t solved the whole problem, of course; your SAs can still manage this and bypass your access admin processes.

And we’ve now got a massive “key distribution” problem. Where should John’s keys be copied to? When he leaves, or changes teams, can we remove all these keys? Can we force key rotation when they get to a certain age? Can all this be automated? Maybe even hooked into your access admin tools?

“John is an Oracle DBA; he’s allowed to login to the oracle servers” is a problem your access admin tools should already be solving; can some form of ssh key management be added so that John’s public key is published to those servers automatically?

There’s a number of challenges that such an automation tool needs to deal with:

Allow a human to have their own unique key (maybe self-generated or system generated), and have that deployed to servers they’re allowed to login to.
Allow a functional account to have multiple keys with command and from (and other) restrictions applied to them. Auto-generate the keys and auto-deploy the private key to the authorized servers.
Handle key rotation so that automation processes don’t break (add new public key to target servers, replace private keys, remove old public keys)
Handle expiration, re-certification, automation, integration…
Manage the GlobalKnownHostsFile so that clients can be told about server host keys, handle rebuilds/redeploys, dynamic host builds, etc.

SSH keys can be used to bypass your existing controls solutions and so need to be controlled. This is an area that hasn’t seen a lot of focus, but I predict a number of regulatory environments are going to start looking at how keys are managed. You can either forbid keys, totally, or manage them.

I recommend trying to manage them!

Single point of truth

Wed, 31 Aug 2016 21:29:44 -0400

Something I’ve been pushing (and this is pretty much a truism amongst anyone who’s looked at “Cloud”) is the idea of automation. It doesn’t matter if you’re just treating the cloud as an outsourced datacenter or if you’re doing full 12-factor dynamically scalable apps. Automation is the key to consitency and control.

So, ideally, this means your automation system is the “single point of truth” for your estate. Whether you use ansible or chef or (saints preserve us) cfengine, your configuration file explicitly defines your target state. You can learn everything from that.

But is this true?

It’s nice in theory but, as is always the case, practice may be different.

Your source of truth may contradict itself.

Now cfengine is easy to see; one promise could say “X is true” and another promise could says “!X is true”. cfengine will complain that these rules don’t converge (assuming anyone reads the logs) and your server is in an unknown state. This is simple.

But there’s a more subtle failure mode.

Let’s say we use ansible to build our environment. The build process calls a sequence of playbooks to take your machine from raw state through to final configuration. So far, so good.

Now let’s say each playbook should be in its own git repo; after all, the playbook that installs and configures apache doesn’t really need to impact the playbook for postfix. It makes sense to seperate out these playbooks into different areas; different teams may be responsible; different access controls can be applied (you don’t want the SMTP team to impact your web servers).

OK, that’s a contrived example, but you can see how it goes; the team building out your Postgres database automation shouldn’t necessarily have the ability to change the configuration of your OpenLDAP servers.

But here’s where things get complicated…

Sometimes there is overlap. Your apache automation may configure the addresses of your single sign on servers. Your nginx configuration may require the same data. If they’re in different repo’s, then how do you ensure consistency?

Your single point of truth (“this is the single signon server”) may not be consistent.

Iteration

There’s no simple answer to this. How you factor your code repositories, how you factor your automation, how you build systems will evolve over time. But be aware; if you define a variable (“single signon server”) for one playbook, maybe it’s also useful elsewhere? Define a global namespace?

Laziness

I spotted this in my own tooling. I have a script that will build my DNS and DHCP configuration. Given an entry in a config file it will build A, AAAA and PTR records for the machine.

I noticed, today, that one of my domains isn’t controlled this way. It has an A and AAAA record that’s hard-coded. I’m sure this will bite me in the bum down the line (when the primary server fails and I need to failover to a secondary). Will I remember this? Or should I fix my automation. The answer is obvious…

Building an OS container

Tue, 23 Aug 2016 19:13:55 -0400

In a previous blog entry I described some of the controls that are needed if you want to use a container as a VM. Essentially, if you want to use it as a VM then you must treat it as a VM.

This means that all your containers should have the same baseline as your VM OS, the same configuration, the same security policies.

Fortunately we can take a VM and convert it into a container. Mostly.

In my lab setup (“lab” be a grandious word; it’s a single machine running too many KVM virtual machines :-)) I’ve created a process for building a new VM via kickstart. I can run virt_build test2 and it will create the LVM volumes, start an install of CentOS 7, patch it, apply my local configs (create my account, install SSH keys, etc etc). A few minutes later I can then ssh test2 and, just like that, I have a new VM.

Many enterprises will have a similar build process; whether it uses PXE to do the initial boot and [Cobbler](https://en.wikipedia.org/wiki/Cobbler_(software%29) to dynamically build the kickstart file, or does it the simplistic way I do doesn’t really matter. The result is an automated consistent build.

Side note: Some artifacts of the automated build process might need to be split into a “build” and “first boot” section; for example a server may register itself to Active Directory for authentication, or to Tivoli for monitoring… these are now “first boot” type activities and can’t be done at build time. But let’s ignore this complexity :-)

The important parts of my kickstart file are:

network --onboot yes --device eth0 --bootproto dhcp --ipv6 auto

rootpw  --iscrypted yeahyeahyeah
authconfig --enableshadow --passalgo=sha512
firewall --disabled
selinux --disabled

zerombr
clearpart --all --initlabel
part /boot --fstype=ext4 --asprimary --size=500
part swap --asprimary --size=512
part / --fstype=ext4 --asprimary --grow --size=1

There’s a %packages and %post section (which creates my account) and so on. This should be perfectly familiar to anyone used to kickstart.

Now the result of my build process is an LVM disk image consisting of three partitions

/boot
swap
/

This, in my case, is /dev/Raid10/vm.test2.

So now we need to take this image and convert it to a docker container.

So we need to make the image accessible to the host OS. Then we can mount it. Then send it to the docker server, where it can be converted to an image.

$ sudo kpartx -av /dev/Raid10/vm.test2
add map Raid10-vm.test2p1 (253:21): 0 1024000 linear /dev/Raid10/vm.test2 2048
add map Raid10-vm.test2p2 (253:22): 0 1048576 linear /dev/Raid10/vm.test2 1026048
add map Raid10-vm.test2p3 (253:23): 0 6313984 linear /dev/Raid10/vm.test2 2074624
$ sudo mkdir /tmpmount
$ sudo mount -r /dev/mapper/Raid10-vm.test2p3 /tmpmount
$ cd /tmpmount
$ sudo tar cf - . | ssh dockerserver docker import - c7test2

$ cd
$ sudo umount /tmpmount/
$ sudo kpartx -d /dev/Raid10/vm.test2

Now this may through up a couple of errors about sockets (in my case it complained about postfix sockets), but we can ignore them.

tar: ./var/lib/gssproxy/default.sock: socket ignored
tar: ./var/spool/postfix/private/virtual: socket ignored
tar: ./var/spool/postfix/private/bounce: socket ignored
....
tar: ./tmp/ssh-IB3AMGEQUc/agent.1873: socket ignored
sha256:374dc7777cfc8c6c2676021dfdd535aac1047a4794a1333814ddcf4936d32b33

If we now look on the docker server:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
c7test2             latest              374dc7777cfc        3 minutes ago       1.185 GB

Notice how large this is, compared to the official docker centos image

docker.io/centos    latest              970633036444        3 weeks ago         196.7 MB

But we now have an OS container that we can run!

$ docker run --name container1 c7test2 /sbin/init &

$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
0e640e27a91b        c7test2             "/sbin/init"        26 seconds ago      Up 22 seconds                           container1

And we can see the container is running a full OS tree

$ docker top container1 | head -4
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                30732               9504                0                   19:49               ?                   00:00:00            /sbin/init
root                30755               30732               0                   19:49               ?                   00:00:00            /usr/lib/systemd/systemd-journald
root                30793               30732               0                   19:49               ?                   00:00:00            /usr/sbin/rsyslogd -n

I haven’t done anything clever with the networking here, but

$ docker inspect container1 | grep IPAddress

tells me this container is 172.17.0.3

So we can now ssh into it!

$ ssh 172.17.0.3                            
The authenticity of host '172.17.0.3 (172.17.0.3)' can't be established.
ECDSA key fingerprint is 32:31:44:a0:9b:61:1b:ff:2f:db:76:ae:a9:a5:36:2b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.3' (ECDSA) to the list of known hosts.

0e640e27a91b$

My OS image is now a (mostly) working container with a funky hostname :-)

What doesn’t work? Well, I don’t seem to have a getty process running on the console. And commands like dmesg don’t work. And I haven’t joined it to my main network (using the default docker0 network). But for all intents and purposes, this is a working OS container.

(For clean-ness we should create a derived container based on this image that automtically runs init and lets the docker command return, but that’s a refinement).

Summary

Now I’ve walked through how I could convert my build process to creating a docker OS container. Naturally you’ll need to modify your own processes to work in your environment, and the more complex the environment the more tweaks necessary. But this is the concept you can follow; take a known good build from your standard build process and convert it to a container.

I’m not really a fan of the “container as a VM” model. I can understand that some people might want to do it and, as previously written, I think that the lines between a container and a VM will blur (at which time the build process will also become easier).

But if you’re going to do it, building a process that maintains consistency between your VM and container builds is a massive step forwards towards controlling the estate and, with automation, can simplify your compliance stance; you only have the one build process to worry about :-)

Using a container as a lightweight VM

Mon, 15 Aug 2016 11:39:58 -0400

In a lot of this blog I have been pushing for the use of containers as an “application execution environment”. You only put the minimal necessary stuff inside the container, treat them as immutable images, never login to them… the sort of thing that’s perfect for 12 factor application.

However there are other ways of using containers. The other main version is to treat a container as a light-weight VM. Sometimes this is called “OS container” because you’ve got a complete OS (except the kernel) here, and you treat it as if it was an OS.

Indeed, this is how earlier container technology was used; on Solaris the zones technology is essentially used to build a lightweight VM. A number of the cheaper virtual host solutions use container technology such as Virtuozo or OpenVZ to present something that looks like a VM.

Characteristics of an OS container

A container used in this way has a number of characteristics that are almost the opposite of what I’ve previously been talking about. So an OS container may have the following characteristics:

Persistent
- Storage is allocated and dedicated to the container. You can treat /home and /usr and /opt the same as you would on any server; reboot the container and the files will remain.
Long lived
- These containers may have uptimes of weeks or months, and even after a reboot the container comes back as it was before
Mutable
- The contents change. Applications may write log files, users may store data, database datafiles may be present…
Accessible
- People may be able to ssh into these things
Manual changes
- If people can login and touch files then we have manual processes.

For me, this sounds exactly like how we would think of a traditional virtual machine (VM). And this means we must treat them like a traditional VM.

Controls

Now in the VM world we have a number of controls that take place inside the VM. You may have things like:

Identity Management
Privilege Escalation
Logging
Change management
Auditing
Backups
Patching
Monitoring
Vulnerability scanning
Intrusion detection
… and everything else!

For an OS container you will need all of these as well. Container technology doesn’t give you any special wins over traditional VM technology. Because you’re running an OS you have to manage an OS.

Now we also have controls around the management of VMs:

Inventory management
- what VMs are present, who owns them, what is their purpose
IP address allocation, DNS, etc
Hypervisor access
Console controls
Control plane access
- Changing the VM size, attached devices, power management
Affinity rules
- Production and DR should not be on the same physical machine
… and everything else!

We’ll need all of these in the OS Container model as well. However some of these controls may look different. It’s easy to control access to the hypervisor, but in a container world the same equivalent would be access to the parent OS. You have to treat those parent OSes as highly privileged, as high as you treat your ESX Hypervisor (for example).

This means building out a lot of infrastructure and tooling to manage these containers; tooling you already have present for VMs.

So why do it?

Since you don’t really gain any of the normal container benefits, why would you use the technology this way? Some of the answers I’ve received to this question are:

Increased application density
- We can fit more OS containers into a machine than we can VMs
Easier for developers
- In a strict development environment we can let dev teams spin up OS containers and experiment in them (to me this points to a failing of existing VM build processes). Perhaps even on their laptops, offline.
Lift and shift migration path
- Build your app into a container, test it, get it working, then migrate it to a public cloud service.

Are there other use cases? If you can think of some then please let me know in the comments!

Conclusion

I commonly hear “you can’t pass audit” type statements with respect to OS containers. I’m not sure that’s true. You can, as long as you have the right controls in place. Since you’re using these containers as a form of lightweight VM then you must treat them as if they are a VM.

All of the existing controls must be in place, and the control layer properly secured. For example:

You don’t allow anyone to create VMs without some checks and balances; you shouldn’t allow them to spin up OS containers without the same controls.
You don’t allow anyone to access the hypervisor; you shouldn’t allow anyone to access to host OS.
You don’t allow unpatched vulnerabilities to exist on VMs; you shouldn’t allow them to exist inside OS containers.

We’re in the early days of OS containers (despite their age), especially when coming to enterprise management of them. We learned how to manage VM farms; we’re going to need to learn to manage OS container farms to the same level of control. Indeed I foresee management planes that will cover both VMs and OS containers in the same way.

Remember that security isn’t absolute; there are gaps in hypervisor security (KVM, Xen, ESX… they’ve all had security gaps in the past). Containers are newer and the kernel exposes a larger footprint and so likely has more gaps. However I can see a day where the line between a VM and an OS container ends up blurring, and the traditional VM itself ends up being reserved for the ability to run different OSes (eg Windows, Solaris, Linux) on the same hardware and OS containers become the predominant way of running multiple Linux installs.

Lift and Shift

Mon, 08 Aug 2016 15:36:52 -0400

A phrase you might hear around cloud computing is lift and shift. In this model you effectively take your existing application and move it, wholesale, into a cloud environment such as Amazon EC2. There’s no re-architecting of the application; there’s no application redesign. This make it very quick and very easy to move into the cloud. It’s not much different to a previous p2v (physical to virtual) activity that companies performed when migration to virtual servers (eg VMware ESX).

Naturally, since you’re running in the cloud, you have great availability and scalability. Job done, right?

Well, that depends on the problem you’re trying to solve.

By performing a “lift and shift” you haven’t solved your resiliency or reliability issues. You haven’t solved your backup issues. You haven’t solved your security issues. Indeed, these issues may become worse, as we’ll see.

Underneath your cloud instance is a physical machine. Physical machines need patching or repairing. This is scheduled downtime that is now out of your control. Your cloud based operating system instance could be shut down and rebooted without any notice. This means you still have to build old-school “high availability” solutions; clustering, fail-over, etc. Because you may not get any notice this HA solution needs to be better automated than you might already have.

Because you’ve just copied your existing solution into a cloud you still have to manage your operating system instances; that means user identity management, audit trails, change controls and so on. However this cloud instance may be (likely will be) outside your security perimeter so now you need to extend your Identity and Access Management systems to that. Worse, you now also have to control access to the cloud provider’s dashboard; who is allowed to spawn new virtual machines, change network definitions, firewall rules?

Security scanning of your environment hasn’t gone away either; you need the same level of control (patching, vulnerability management, etc) as you had before.

Your cloud provider may give you nice resilient disks; that’s nice! But resiliency isn’t the same as a backup solution. You still need to backup your files (e.g. for 7 or 10 year retention to meet audit requirements).

All of this may work out more expensive than doing it in house. People look at Amazon and say “wow, that’s cheap! So much cheaper than my company can provide a virtual machine for!” And the up-front number is cheaper. But then you add in all the additional tools that you use (I&AM, vuln scanning, logging, network traffic, backups, patching…) and notice your in-house cost has all that bundled in and it can easily turn out that your cloud instance works out more expensive.

So with all these negatives, why would you “lift and shift”?

Well, it’s not a once size fits all problem. A large mega-corp running 30 datacenters should be able to provide virtual servers cheaper than Amazon. But a small company that only needs 100 servers? Or only has a couple of datacenters? Well now the equation isn’t so clear cut. It may well be worth effectively outsourcing the datacenter to a cloud provider. Why go to the hassle of maintaining physical servers, needing physical security guards, staff to manage the HVAC systems, NOC teams to replace failing kit…

Having your kit in the cloud can also help you migrate to a more modern architecture; once you’ve got an API driven infrastructure it becomes easier to look into building apps and environments that can start to make use of auto-scaling, micro-services, resilient architectures, 12-factor apps…

Conclusion

“Lift and shift” is a viable model of using cloud computing, but it’s not a panacea. You have all of the management challenges of running in your own datacenter (or co-located datacenter); you need to manage your servers, build HA solutions, replicate data, have backups etc etc. It may, initially, even cost you more money than you expected.

But once you are there you can start to take advantage of other features; want to use Apache Spark? Amazon already have this pre-built; you don’t need to engineer that yourself. Want to make use of multiple datacenters, close to your customers? They have plenty of points of presence to assist in that.

Don’t go looking at the problem as “We need to be in the cloud! Let’s lift and shift”. Instead look at the use cases; would it make sense to re-architect and rebuild your app to be more “cloud native”? Maybe even something simpler… perhaps you’ll find you don’t need to run your app 24x7; only spin it up one hour per day…

“Lift and shift” is one of the crudest methods of getting into the cloud. But it doesn’t make it a bad way of doing it. It’s just another tool in the toolbox.

Persistent data

Thu, 28 Jul 2016 08:35:00 -0400

In this glorious new world I’ve been writing about, applications are non persistent. They spin up and are destroyed at will. They have no state in them. They can be rebuilt, scaled out, migrated, replaced and your application shouldn’t notice… if written properly!

But applications are pointless if they don’t have data to work on. In traditional compute an app is associated with a machine (or set of machines). These machines have filesystems. We can write data there. If we want to share data between machines we can use something like NFS. It’s very easy to persist data.

In our new dynamically scalable app migrating world we don’t have this.

So where do we store our data?

The standard answer is to use an external datastore, such as MySQL or CouchDB or an object store (typically presented with an Amazon S3 compatible API, even if not actually using Amazon S3). Your application doesn’t persist any data; these resources are attached (or bound) to your app so they can be used.

Even for users of databases this may require a change in behaviour; you might write out all your important data into the database but write out logs and performance data to the filesystem. You can’t do that any more; everything you want to keep needs to stored in the database or S3 store. And that requires a code rewrite.

Persistent files

But I’ve never been a fan of that. Why can’t we treat a filesystem as if it was another attached resource? This data could also be shared between instances of the app.

With docker we have some of this ability with the -v flag. Let’s create a directory and share it across two running instances:

In terminal window 1:

tty1$ sudo mkdir -p /export/myapp
tty1$ docker run --rm -it -v /export/myapp:/myapp --name inst1 centos 
[root@cd0c4a2b0055 /]# ls /myapp
[root@cd0c4a2b0055 /]# echo world > /myapp/hello
[root@cd0c4a2b0055 /]#

In window 2:

tty2$ docker run --rm -it -v /export/myapp:/myapp --name inst2 centos
[root@988151662ef1 /]# cat /myapp/hello 
world
[root@988151662ef1 /]#

We can shut down both containers, and they’ll be destroyed (because of the --rm flag) and the data will persist on the host

[root@cd0c4a2b0055 /]# exit
tty1$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
tty1$ ls /export/myapp/
hello
tty1$ docker run --rm -it -v /export/myapp:/myapp --name inst3 centos
[root@7ed3b5e955f6 /]# cat /myapp/hello 
world

So here’s an easy way of persisting data in a manner that developers are already used to.

It’s not that easy

There are a number of problems with this model. First and most important is data security. Because the data is present on the host then anyone with access to the host might be able to read this data. This is why I’ve recommended that production container servers should treat the parent OS with high access restrictions; as restrictive as your hypervisors in a traditional VM.

There may be problems with SELinux or other labelling systems; the directory created didn’t have any of the right labels so if the OS was set to enforcing mode then access to this directory may be rejected (indeed I had to do a setenforce 0 for these tests).

We’ve only looked at a single host; in a real world environment your app may spin up and down across dozens of different servers. So you may need your persistent datastore to be coming from an NFS server or similar. You might just mount that at the host level. Docker also allows for different backends to be used with the -v flag; it can talk directly to an NFS server, for example. That’s pretty powerful!

At lot of this docker functionality is documented in a tutorial.

You also need your orchestration tool to be able to support this configuration and start up your containers with the right flags. Mesos, for example, supports persistent volumes; Kubernettes supports similar.

Beyond docker

There’s nothing that says this has to be docker only; for example, if you use systemd-nspawn to manage your containers then there is a --bind option to do similar stuff.

Even Amazon are in this game with Elastic File System which presents your storage as an NFSv4.1 accessible filesystem that you can mount onto your EC2 server. This means you can take the persistency out of your AMI and put it into EFS; the result is something very similar to the docker examples earlier.

Summary

We don’t have to use storage tools such as S3 to keep to a 12 factor design. It’s perfectly possible to keep your standard filesystem semantics and keep your application layer immutable and all the rest of the goodness.

This can even make operations easier; that same persistent volume could be shared read-only with an “operations application”; your operate team can read the logs, analysis performance statistics on demand (only spin up that app when needed) without needing to access the production application container.

Man in the middle attacks

Sat, 16 Jul 2016 11:26:56 -0400

A fair number of security advisories mention Man In The Middle (MITM) attacks. It’s quite an evocative phrase, but it’s a phrase meant mainly for the infosec community; it doesn’t help your typical end user understand the risks.

So what is a MITM attack, and how can I avoid becoming a victim?

Before we get into technology let’s look at something we all know about; the boring snail mail postal system. You write a letter, stick it in an envelope, drop it in a postbox and then sometime later it appears at the recipients house.

In that travel many people may touch the letter; the postman who takes it out of the postbox; all the people at the local post office who sort it into local delivery or forward it to the regional hub; the sorters at the regional hub and many other hubs, to the recipient’s local post office. If you send a postcard then anyone at these places could read your words, or even change them. All of these people are potential MITM attackers; they sit in between you and your recipient and can do nasty things.

So we put stuff inside envelopes and hope no one steams open the envelope and reseals it afterwards. This isn’t even a new problem, and is one reason why [Wax seals](https://en.wikipedia.org/wiki/Seal_(emblem%29) were popular to provide evidence of tampering.

So what does this have to do with the internet and technology?

When you talk to a webserver the data you send travels somewhat like your traditional letter. It goes from your PC to your local router, from there to your ISPs router, then across various other routers to the webserver’s local gateway and then to the server. At each point the traffic could be snooped upon or modified. Even worse there are effectively invisible points in between (switches, hubs, bridges etc) where traffic can also be spied upon.

Now in reality the majority of this “backbone” is pretty secure. Major ISPs, just like your post office, really try hard to stop traffic from being stolen. But just like postal workers have stolen, it is possible for this backbone infrastructure to be compromised; even from the outside.

Slightly annoyingly, some ISPs deliberately MITM consumer connections to insert adverts into the browser; Verizon have added tracking cookies on their mobile network.

Depending on where you are, ISP level MITM attacks may not be a great risk. (At least if you ignore FBI, NSA et al!). But some places may be more risky; China and North Korea can be pretty much guaranteed to be MITMing traffic.

But if this was the only problem then a MITM attack wouldn’t be big news. Fortunately the solution to the bigger risk can also help here.

A bigger risk

The greater risk is closer to you; how you connect to the network.

Are you connecting from an office? If so then you are likely to be connected to a switch, rather than directly to a router. In the old days this would have been a hub and every one of your co-workers on the same hub could see your traffic. With a switch it’s harder for them to see it… but some switches can forced to drop back to hub mode. Or your co-worker could send proxy ARP requests to make your machine think they are the default gateway server. Or even tamper with the wiring!

Do you have to go via a proxy server to reach the internet? That proxy can see your traffic and so any admin on that machine could see it.

Are you at home sharing your network with flatmates? They might be able to see your traffic.

The traffic between your router and your ISP is probably secure (e.g encrypted by DOCSIS cable modem standards, or via dedicated lines), but is your router or cable modem secure? A lot of home routers have security holes or backdoors, even as simple as having a default admin password and the admin user interface being exposed to the internet.

The server you talk to also have some of these endpoint risks… but major players (Google, Microsoft, Twitter, Facebook, etc etc) work hard to ensure their links to the ISP are secure. Small self-hosted websites (this one!) may be at risk (do I trust linode or Panix to not break my traffic?)

Then there’s the really big risk case… free public wifi. You might think that it’s nice to be able to go Starbucks and use their WiFi. But are you using their WiFi? Or someone else’s WiFi that pretends to be the Starbucks access point?

This WiFi risk is, in my opinion, the greatest of the MITM attack vectors. You can be sitting, sipping your coffee, and having all your internet traffic being inspected. Passwords you type in might be taken. And all this is automatic. It doesn’t require “l33t hax0r sk1llz”; you won’t see a dingy drugged out looking guy furiously typing on his keyboard; it’s invisible.

So how do I defend myself?

Ensure your router is secure; disable any external admin interfaces, change default passwords, apply the latest firmware.

Encrypt your traffic where possible.

If you go to a web site, get into the habit of typing https:// in front of the URL. Perhaps consider a tool such as HTTPS Everywhere to help with this.
If you have configured your mail client to connect via IMAP or POP then make sure the use encryption options are ticked.
If you use a chat application then turn on encryption (ideally full end to end encryption, if supported, but even encryption between you and the chat server is better than no encryption).

Do not let your machine connect to public WiFi points automatically; yes, it’s nice to say “automatically connect to Starbucks” but this allows people to create a fake Starbucks access point anywhere and your phone may connect to it, even while still in your pocket, and any unencrypted communication may be stolen… even when you’re nowhere near a Starbucks!

If you can, consider using a VPN. This can automatically encrypt all your traffic. Even with https and encrypted connections a MITM attacker may still be able to learn some information (e.g. you did a DNS lookup of www.myexamplebank.com; it’s possible that’s who you bank with!). A VPN can hide even this information.

Use a unique password for every site; even if you’ve tried hard to protect against MITM attacks some information may leak out. If someone manages to steal your Facebook password then they can’t use it to login to your bank. A password manager like 1Password can help here by generating complicated unique passwords. (This is a good thing to do anyway for more than just MITM reasons!). Personally, I use KeePass in its various forms.

Anything else?

I’m sure I’ve missed out some other ways MITM attacks can happen, and ways we can defend ourselves. Please drop me a comment if you have any advice that can be added!

There's a hole in my security bucket

Thu, 07 Jul 2016 12:05:45 -0400

In my spare time I’ve been playing on Unix StackExchange. And I’ve found the old song There’s a Hole In My Bucket going through my head. It’s a conversation between Henry and Liza; Henry has a problem and is asking Liza for help.

In summary:

H: There's a hole in my bucket
  L: Mend it!
H: How?
  L: With straw.
H: But it's too long
  L: So cut it
H: How?
  L: With an axe
H: But the axe is blunt
  L: Sharpen it
H: How?
  L: With a stone
H: The stone is dry
  L: So make it wet
H: How?
  L: With water
H: How do I get the water?
  L: In a bucket
H: But there's a hole in my bucket

As problem solvers we’re typically in the Liza role. Someone comes to us with a problem that seems simple (you have a hole in your bucket? Well, fix it!). And it seems frustrating to have to go through really basic remediation steps (surely you know how to cut straw!) and it’s only 10 times around this loop that you start to learn what the real problem is.

This can be even more frustrating if the communication is high latency (e.g. the other person is in a different timezone, such as between New York and Singapore; each part of the conversation could take a whole day!). What should be a simple problem now takes a week to solve.

In an attempt to solve this you write detailed documentation; you try to foresee all possible failure modes and write them down, create a decision tree with steps to follow. And thus begins the disconnect between “engineering” and “operations”. If this goes too far then the ops teams will refuse to support stuff that isn’t fully documented, leading to 100 page handover docs. Ops turn into deskilled button pushers.

This can now lead to security problems; the operations teams don’t fully understand what they are doing, make the wrong choices, take a path of the decision tree that almost but not quite matches their problem and so implement the wrong solution (they put the straw in the bucket at an angle so it doesn’t fill the hole properly; water leaks out).

(This is one reason why “reliability engineering” is becoming popular; an attempt to reverse the trend. Take the button pushing out of the ops role, re-skill the role to the level it should be).

As problem solvers we need to listen and engage more with our operations teams; we need to remove barriers between teams. This isn’t necessarily a “devops” model (in some cases there are regulatory separation of duties requirements that prevent this), but higher bandwidth communication channels, co-located engineering/operations teams, shared responsibility for problems and so on. Encourage a collegiate atmosphere between the teams, rather than an adversarial one which can form when barriers are created (finger pointing between ops and engineering is a common failure mode).

Now we can make sure the right solution is applied to the problem, and the security hole isn’t opened… hopefully!

Next up: how you are the security Itsy Bitsy Spider trying to climb the drain pipe of security and the rain drops are the attackers washing away your work.

See me present!

Fri, 01 Jul 2016 16:05:48 -0400

Sesh Murthy from Cloud Raxak asked me to co-present at Cloud Expo NY June 2016. I’ve never done such a thing before, so this was a big deal for me.

I put together a base presentation that Sesh modified.

The video of this is now on YouTube. My part starts at 8m30, and there was a little Q/A at the end (31m35).

“Enjoy” watching me do my first ever public talk!

I’m a little embarrased…

Container Identity

Mon, 27 Jun 2016 12:06:21 -0400

Containers and other elastic compute structures are good ways of deploying applications, especially if you follow some of the guidelines I’ve made in other posts on this topic. However they don’t exist in a vacuum. They may need to call out to “external” services. For example, an Oracle database, or Amazon S3, or another API service provided by other containers. In order to do this it needs to authenticate to that service. That typically means requiring some form of authenticator (e.g a username and password, or a userkey and secret key).

Now in a traditional non-elastic environment this can be done at application deploy time; an operate person can login to the server and supply the necessary credentials. If you’re more advanced then you might use a password vault (e.g. CyberArk). These work because the operating environment is pretty static. In the case of CyberArk they require an agent to be installed on the client machine, registered, and authorised to the vault. Clearly neither “ops supplying credentials” nor “register agent” type solutions are very elastic friendly! So we need a different solution.

If you look around there are various products that try to solve this; a common one is Hashicorp Vault. This has some clever routines built into it, and was clearly built with an API-first model in mind. One clever feature is that it can create time-limited credentials, so if your app needs to login to MySQL then it can ask HashiVault for a username/password, the vault will create them in the database at that point in time, then delete them later. Clever!

Unfortunately it doesn’t address the question of how your application authenticates to the vault in the first place. We haven’t really solved the problem with this technology; we’ve just rewritten it.

Another solution I’ve heard a few places attempt is kerberos based authentication; however this doesn’t really solve the problem, either. How does your application get the initial TGT to be able to get the service keys it needs? If you use a keytab file, how does that get into the app? No, including that in your app source tree is not a suitable solution :-)

So what can we use as an identity anchor? When your app spins up and migrates around containers, when the network itself may be NAT’ed, when the contents of containers may vary as applications are upgraded… what can we use to anchor the identity so your application can authenticate to other services?

Viewpoints

I’ve found three ways of looking at this problem.

Seed credential

This the most obvious version; This may be your keytab file; if this can be provided securely to the app then the application can now authenticate to the Kerberos infrastructure and carry on. It may be an authentication token to a vault (eg HashiVault) from where it can then get the necessary credentials necessary for further services. How this is done will be technology specific. With something like docker you may be able to provide a persistent backend data store with that credential and this can be shared between the containers running the application. Not all container engines have this persistency layer so other solutions may be required.

Provided by the runtime environment

This is the solution provided by Cloud Foundry. With this information can be stored inside the “cloud controller” and passed onto the application in the VCAP_SERVICES variable. It’s all handled by the orchestration engine. Unfortunately the CF solution stores all this data in plain text and passes it in environment variables. The concept seems reasonable but the implementation is weak. Of course the values provided here may be the actual credentials needed, or a seed credential for vault access.

Have a trusted identity assertion

Can we identify the app based on the execution environment? Perhaps the orchestration layer could generate a “signed identity” (perhaps a client side SSL cert) and supply that to the app when it is instantiated. This could be used as a seed credential. Or maybe we can run a daemon process in the parent OS that the client can call out to; we can make use of the orchestration knowledge (e.g. the state maintained by docker, or a Cloud Foundry DEA configuration file) to identify the calling app; effectively this daemon would assert identity on behalf of the app.

Putting it together

These three viewpoints can overlap; for example the identity assertion tool could talk to a backend datastore that can provide encryption keys (eg an RSA key). The public key could be exposed to the world and so the contents that cloud foundry normally stores in VCAP_SERVICES can be encrypted; the identity assertion process would ensure that only application A could get RSA keys for itself and not for any other app. The credential then decrypted could provide access to a vault.

Of course there are implementation challenges in all these options; if you create an encryption store then how do you ensure that the keys remain secret, or have the ability to rotate keys? If you provide a persistent backend store then how do you ensure the contents won’t be exposed elsewhere? If you use a CF type solution then how do you solve the plain-text problem?

At the end of the day your orchestration layer becomes a critical component in allowing an application to assert identity. The solution you build needs to be resilient, reliable and strongly protected. Of course this should apply to the whole orchestration layer, anyway!

Network Microsegmentation

Mon, 20 Jun 2016 13:12:42 -0400

A major problem many environments have is a lack of real network control inside the perimeter. They may have large hard border controls (multi-tier DMZs; proxy gateways; no routing between tiers), but once inside traffic is unconstrained. This is sometimes jokingly referred to as “hard shell soft center” network design.

If you’re lucky then your prod/dev/qa environments may be segmented. More likely there’s no restriction at all; dev programs may accidentally talk to a prod database. Oops!

And, of course, this “open network” enables an attacker who manages to gain entry to the network the ability to pivot and attack other unrelated servers.

The question to ask is whether we can control things at a deeper layer. Can we determine what applications need to talk to each other? In a microservices model, can we control which service each application component needs to talk to?

In our existing environments this is hard to do; things have grown haphazardly and we have no real visibility around where our “command and control” (C2) servers are; your vuln scanning tool, your backup servers, your CMDB collection tool, your automation tool… these servers may not have been placed in any consistent location and mapping out the connections at scale is not easy.

At best we might be able to restrict “consumer banking” servers from interacting with “investment banking” servers; restrict the ATM network from seeing the HR servers… a form of “macro segmentation”.

In the new “cloudy” world, however, we get a chance at revisiting this. I’ve previously written about automation of application instantiation; hands off deployment of servers, containers, applications. We can take this further and do hands off deployment of networking as well. “Software defined networking” allows us to build and modify network constructs in a programmatic automated manner. We can build logical constructs for different applications, for services, and define rules for communication between these constructs. We can associate these rules with the app deployment so that routing, packet filters (eg iptables) and the like can be updated as part of application orchestration.

Egress rules can also be applied; your app configuration indicates what services it needs to talk to. Your orchestration layer can build out egress iptable rules outside the container (nothing that happens inside the container can change this), and a set of logging rules can be set up so that bad traffic is trapped and reported on.

These logs can be sent to your SIEM system and monitored the same as any other security log. If you now see an application container start to make a lot of port 22 connection attempts then you know something has gone wrong and you can take action (e.g. shutdown the container or track back where the attack came from; analyse the attack vector and fix the code). Of course the rules will have prevented pivoting, but now you also get a deep level of visibility of what is happening inside your network, not just at the border.

Summary

Tools are still developing in this space; there are networking vendors that can handle routing via BGP configurations (your app joins a BGP group and the centralised rules determine what services are visible), combined with kernel iptable/ipset rules configured via the BGP rules. Some of these are drop-in replacements in the OpenStack setup and extends the “security group” construct.

The Cloud Foundry PaaS also has security groups associated with applications that configure egress rules per application component.

If you use Amazon then you can configure “virtual private clouds” and segment networks.

There are multiple ways of doing network segmentation, but taking advantage of application redesign and new automation that needs to be built out to handle a cloudy environment means we can avoid mistakes of the past; increase security in a practical and controlled manner; limit exposure when (not if, when) an attacker gets in; gain greater visibility of your network setup.

Add in something like Google’s BeyondCorp perimeterless network design to handle the physical layer and we’ve gone a long way!

Using Containers Securely in Production

Mon, 13 Jun 2016 19:38:31 -0400

This is the content of a presentation I put together for Cloud Expo NY 2016. The final presentation had a lot of this ripped out and replaced with stuff from my co-presenter (Sesh Murthy from Cloud Raxak), because he had information he wanted to present as well and we only had 35 minutes. The resulting presentation was, I think, a good hybrid.

This is the original story I wanted to tell. You may recognise some of this content from other blog posts I’ve written :-)

For each section I’ve added in italics my thinking and what sort of thing I’d have talked about.

I attach the LibreOffice Impress presentation for those who want to see the original.

What is a container?

Shared kernel
- Segmented view of resources
- Separate process ID space
- Separate filesystem mount space
- Separate IPC memory segments
- Separate view of the network
- …
Multi-platform
- Linux VServer (from 2001!)
- OpenVZ
- AIX WPARs
- Solaris ~Containers~ Zones (from Solaris 10, 2005)
- Linux containers (LXC, cgroups, etc)
- Docker
- Warden
- …

The thinking here is that there’s a lot of discussion about containers, but there’s different ways of thinking about containers. I wanted to focus on the lower technology layer, rather than the packaging, at this point to show that this type of technology wasn’t new. I’d used the VServer patches in 2001 to create “bastion” hosts on my firewall.

Container as a VM

Let’s look at the worst case scenario. A container may have:

A cut down (maybe!) version of the OS
- Glibc, bash, web server, support libraries, tools
- Maybe different versions than those running on the host!
Filesystems
- Private to the container?
- Shared with the host?
Access to the network
- Bridged to the main network?
- NAT?
Processes that run
Network listeners
User logins

This sounds like a whole operating system to me. The only difference is the shared kernel.

So, in the worst case, we need to control this the same as the OS. And deal with the unique challenges.

I call this “the hard way”. You can treat containers like this; indeed Solaris zones are exactly like this. But effectively you’re creating a form of “virtual machine”, and so need to control it that way.

Challenges with this model

	Virtual Machine	Container	New threat
"Parent Access"	Hypervisor attack; one VM may be able to use hypervisor bugs to gain access to another VM	Parent OS attack; process may be able to escape the container	More people may have access to the host OS than to the Hypervisor; more threat actors. Shared kernel is bigger than hypervisor; bigger attack surface Kernel may allow dynamically loaded modules; variable attack surface!
"Shared resources"	Rowhammer Noisy neighbour Overallocation ...	Mostly the same	Resource separation is now in the kernel; not as well segregated; not all resources may be segregated
"Unique Code"	Each VM may run a different OS, different patch revisions, different software	Each container may run different software versions, different patch levels, different libraries	Scaling issue; we can run many more containers than VMs
"Inventory management"	What machines are out there? What OS are they running? Are they patched? What services are they running?...	Mostly the same	We typically have strong controls as to who can create new VM's but we allow anyone to spin up a new container 'cos it's so quick and easy!
"Rogue code"	Is the software secure? Acceptable license terms? Vulnerability scanned?	Mostly the same	Anyone can build a container; it may have different versions of core code (eg glibc) than the core OS. Developers may introduce buggy low-level routines without noticing

If you are going to use a container as if it’s a VM then you need to apply all your existing VM level controls to the container; you control VM creation, you’re going to need to control container creation to the same level! Otherwise you run the risk of “Shadow IT” with unknown software and potential vulnerabiltiies (shellshock, heartbleed etc).

Why are you using containers this way? Are you just using them as a packing construct?

I don’t like spending a lot of time on this page, but people typically lean forward and pay attention. Maybe it’s the wall of text that makes them think there’s good content! The point here is to emphasise “the hard way”. And if you’re using containers as a VM technology then you might want to rethink your approach.

What makes a container different?

Very dynamic
- Harder to track, maintain inventory
- If you don’t know what is out there, how can you patch?
- How many containers may suffer shellshock, heartbleed or CVE-2015-7547 (glibc) issues because we don’t know where they run?
Technology specific issues
- Docker daemon running as root?
- SELinux interaction
- Root inside a container may be able to escape (eg via /sys)
Kernel bugs
- Ensure your parent OS is patched!

What is now highlighted is that the existing solutions may not scale properly. What works for a 1,000 (or even 10,000) VMs won’t work for 100,000 or millions of containers.

Can your solutions deal with containers spinning up, running for a few minutes, then shutting down again? Probably not.

This helps distinguish a container from a VM, especially the scaling factor and dynamic nature

The Easy way

Think of a container as

Transient
- There is no persistent storage
Short lived
- Anywhere from microseconds upwards
Immutable
- You don’t change the running container; you build a new image and deploy that
Untouchable
- You NEVER LOG IN!
Automated
- If you can’t login, you better automate the builds

Now we can start to focus on what matters; the application. Don’t allow people to create containers; have an “application delivery platform” that does it for them.

Short lived also means it has a drop-dead date; this may be minutes, hours, weeks… Have a chaos monkey do it, or have a risk-based evaluation.. but a container should die. They’re not persistent.

The “immutable” and “never log in” parts of this can be hard to swallow. It means a change in AppDev processes (closer to 12-factor applications; it means the app needs to instrument and log stuff better). But this is key to elastic compute.

Minimize attack surface

Basic hygiene rules:

Harden the kernel
- You don’t have to use the vendor provided one
Ensure the OS is patched
Integrate into your CI/CD process
- No manual startup of containers
- Only those that passed testing, vuln scanning, etc can be run
- No external code without approval
- Standard code hygiene (provenance, licensing, etc)
Automate, automate, automate!

A hardened kernel doesn’t mean “create your own patches”. You can take a kernel direct from www.kernel.org and configure it with a minimal set of options built in. e.g. turn off dynamically loadable modules (they open a large attack surface); only add in drivers necessary for your hardware (easy if you always deploy to a VM platform); don’t add features you don’t need.

Container life cycle

The contents of a container are also important

In general there are a number steps:

Build good containers
- Known content, internal repos
Deploy containers
- Orchestration layer, autoscaler, track deployments
Scan existing containers
- Keep on top of new found bugs
Replace bad containers
- Replace stuff known to be broken

We’ll go through each of these steps in the next few slides

Build good containers

You already know how to do this… extension of your existing processes!

Don’t create a whole copy of the OS to put inside the container
- If you need an OS image then use a common base (the same base as you use for traditional compute?). Don’t have a RedHat base, a Debian base, a Ubuntu base… stick with one image
- If you use golang, how much do you actually need inside the container? A few /dev files?
Don’t pull code direct from the internet.
- npm “left pad” chaos
- Use internal source repositories
- If using docker repos, use internal ones
- Curate the content that goes into them
Code scan
- Source code scanning
- Binary object scanning
- CVE detection, OWASP best practices
Automated deployment
- Final image push to a registry; deployed to dev (automated)
- Promoted to QA/UAT
- Promoted to prod

Really this is meant to be an extension of current best practice that people are already using for code creation and deployment, but extended to handle the container build process. So don’t pull docker images from a public registry; use your own private one… in exactly the same way that you shouldn’t pull code live from the internet but have an internal source repo.

You can then start to automate things; e.g. have a Jenkins job that’s triggered from a code checkin that does the code scanning, builds the image from the internal repos/registries, validates it passes all checks, pushes into the dev registry. Consider that image immutable and your code promotion becomes easier; just promote good images.

Deploy containers

Orchestration/Automation needs to

Deploy only containers from registry
Track which containers are running where
- Effectively maintain “inventory” of code running

Key to deployment at scale

Auto scaling
Auto repair
Blue/Green deployments

This is a small slide, but important. Your orchestration layer (Kubernettes, Cloud Foundry, Swarm… whatever you use) needs to be able to track and report on what images are running, and how those images were built. We’ll see later why…

Scan Existing Containers

Target state isn’t static

New CVEs found

Libraries
Base OS image
Compiler
JVM
…

Keep scanning everything in the registry. When new bug (eg shellshock) found we can identify every bad image

There’s a new OpenSSL bug every other month. Just because your container was clean when you built it doesn’t mean it’ll remain clean. So we need to keep scanning these container images in case a new bug is found anywhere in the application stack.

Replace bad containers

Your scanning detected a problem

Your orchestration layer identifies where these images are running

Risk based approach

This is where the orchestration layer importance comes in. It’s all fine and good knowing the “myapp” image is bad, but if we don’t know where it is running then you can’t replace running images.

How quickly you replace it is up to you; if you have an internet facing shellshock then you might want to cycle your containers pretty quickly, but in an internal public-data service may wait until the next container recycle.

Nothing new!

You’re already doing this for regular compute.
Right?
- Right?
Nasty secret… it’s where we want to be, but large organisations take time to pivot

Use this to fix existing procedures

Same solution can work on VMs

Internal IaaS
Amazon AWS via AMI images

There’s nothing really unique about containers, here! You can build OS images the same way and build an orchestration layer on top of it and manage virtual machines the same way! Have we been managing VMs wrong for the past 20 years?

Technology

Nothing here is technology focused
- Process is the key
Only need to care about low levels if building your own
When using docker/Apprenda/CloudFoundry/$XYZZY treat it like any other vendor product
- Track upstream bugs, patch
- Follow best practices
- 1,000s of web pages on “how to docker”
Control the environment
- Automation
- Hands free deployment
- You need to do this, anyway!

That’s why this was sub-titled “It’s the process that matters, not the technology”.

Big wins

How many licenses are you using? You know because your automation tools will tell you what is running
A CVE has been found; you know what images are vulnerable and where they are running
- Your image is more lightweight so may not be vulnerable in the first place!
Patching doesn’t exist; you just redeploy
Change management is simpler
- No need for “tripwire”; no one can login to change contents
- No need for central IDadmin; no accounts to manage
Financial regulatory controls become easier
- But may require educating auditor and regulators on the “new world”
Controls end up being pushed to “code management”
- Which you need to control, anyway!

And another benefit… if your app can be delivered this way, you can also start to deploy VMs this way. A Linux OS without any form of login? Let’s see someone brute-force ssh bad passwords that way :-)

Teaching regulators and auditors about this new world is going to be interesting; they’re gonna freak out!

Conclusion

Use the right tools for the job
- If you want a VM then use a VM
If you treat a container as if it’s a VM then you need to control it as if it’s a VM
- Heavy process
- Not always scalable
Must control containers
- Avoid Shadow IT
- Must not allow a “wild west”
Not all tasks are suitable for containers

Reminder

Container security is weaker than VM security; the kernel has a large surface area to be attacked.

VM security is weaker than physical machine security; the hypervisor provides an attack surface

Network security is weaker than air-gap separation.

Air-gapped machines are weaker than a server covered in cement and dropped into the bottom of the ocean.

Your security stance depends on your needs and risk evaluation. Pick the technology that is right for you and use it appropriately.

I was told to always end a presentation with a joke

What made me start thinking about security

Sun, 12 Jun 2016 13:14:18 -0400

Back in 1984 I thought I was pretty good at writing programs for my BBC Micro. I could write BASIC programs that worked; I was learning 6502 assembler. I could hack on programs, break copy protection. I definitely knew more than my teachers.

But my brother was able to break my code.

For example, I wrote a simple “football” program for him. The idea was that he’d select two teams and the game would simulate a match and generate some scores.

The opening screen would present a list of teams and let him select them. I thought it would go something like this:

What my brother did, to my surprise, was something like this:

That was a new thought for me; people entering unexpected input! But a good lesson to learn at such an early age. Always validate the input.

This lesson really came into its own in the early days of the ‘net; a co-worker was really proud of the CGI program he wrote that would allow stored files to be displayed. He was less happy when I passed ?file=../../../../../../etc/passwd as a parameter :-)

Today a number of these lessons have been formalised into patterns by organisations such as OWASP top ten project. Obviously, today, threats are so much more advanced than the simple problem my brother found 30 years ago. But sometimes just reminding people “don’t trust the client to send good data” is essential; people can be focused on making sure they get an A+ SSL rating but then allow SQL injection attacks through blind trust of supplied input.

Building a small docker container

Sat, 04 Jun 2016 13:45:31 -0400

In previous posts I’ve written about small containers; don’t bundle a whole OS image with your app, just have the minimum necessary files and support.

The Go language makes it easy to build a static executable, so let’s use this for an example:

$ cat hello.go
package main

import "fmt"

func main() {
  fmt.Println("Hello, World")
}
$ go build hello.go
$ strip hello
$ ls -l hello
-rwxr-xr-x. 1 sweh sweh 1365448 Jun  4 13:48 hello*

We can use this as the basis of a docker container (I’m using “docker” here because it’s a very common technology that’s used by lots of people):

$ cat Dockerfile                 
FROM scratch
COPY hello /
CMD ["/hello"]

$ sudo docker build -t hello:go .
Sending build context to Docker daemon 1.369 MB
Step 1 : FROM scratch
 ---> 
Step 2 : COPY hello /
 ---> 921eb0866d50
Removing intermediate container e518d193b269
Step 3 : CMD /hello
 ---> Running in 0080bf495dd0
 ---> 0480444b8e0a
Removing intermediate container 0080bf495dd0
Successfully built 0480444b8e0a

$ sudo docker run --rm -t hello:go 
Hello, World

$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
hello               go                  0480444b8e0a        58 seconds ago      1.365 MB

That’s a nice small container! Let’s see what it consists of

$ mkdir EXTRACT         
$ cd EXTRACT
$ sudo docker save 0480444b8e0a | tar xf -
$ for a in */layer.tar
do
tar tvf $a
done
-rwxr-xr-x 0/0         1365448 2016-06-04 13:48 hello

So the whole container image contains just one file.

Compare that to a more traditional container built with a Debian base:

$ cat Dockerfile                  
FROM debian:jessie
COPY hello /
CMD ["/hello"]
$ sudo docker build -t hello2:go .
Sending build context to Docker daemon  1.37 MB
Step 1 : FROM debian:jessie
Trying to pull repository docker.io/library/debian ... jessie: Pulling from library/debian
23286f48d129: Pull complete 
7a4c9a4d5e7a: Pull complete 
Digest: sha256:2ca1d757fce75accad6ff84339c3327c7aa96ad6e7b7d6fdde22b2a537fac703
Status: Downloaded newer image for docker.io/debian:jessie

 ---> 7a4c9a4d5e7a
Step 2 : COPY hello /
 ---> 53afe819cb68
Removing intermediate container b502f926d128
Step 3 : CMD /hello
 ---> Running in 47c55d108c18
 ---> 047bf1fe5c58
Removing intermediate container 47c55d108c18
Successfully built 047bf1fe5c58

Yikes, we’ve just pulled in external content. That’s a “no no” from a controls perspective.

The container is now a lot bigger:

$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
hello2              go                  047bf1fe5c58        About a minute ago   126.4 MB
docker.io/debian    jessie              7a4c9a4d5e7a        11 days ago          125.1 MB

$ mkdir EXTRACT 
$ cd EXTRACT
$ sudo docker save 047bf1fe5c58 | tar xf -
$ for a in */layer.tar
> do
> tar tf $a
> done | wc -l
8249

8,249 files are now present in this container! That’s a larger footprint for an attacker to exploit; if an RCE is found in your ‘hello’ app then they now have a full shell environment to exploit. It’s also more complicated to patch and manage; if a bug is found in the underlying ‘jessie’ image then that needs updating and any image that depends on it needs to be restarted, as well.

A PaaS such as Cloud Foundry tries to make the layering a bit more invisible; the underlying “OS image” is part of a stem cell. Your application and dependencies are part of a droplet and are created using buildpacks. In some ways this makes it easier to manage; the team managing cloud foundry can update/replace the stem cells and your app shouldn’t notice; it just gets restarted with the new stem cell. However it does mean there’s a full OS image hiding inside the container, which could be used by an attacker.

Conclusion

Now, admittedly this is a simple case; no networking, no web listener, no additional content. But we can see how simple it is to build a container with no hidden “layered” dependencies. You need to spend more time working out your minimal dependencies first (statically compiled languages such as “go” make this easier, but it can be done with traditional binaries such as httpd, just by following the library dependency chains).

The end result is a smaller, more secure image. There’s less to assist attackers (no unnecessary files so harder to get a shell) and simpler to maintain (smaller dependencies).

It also helps segregate the application from the OS by making the dependencies explicit; switching from a Debian base to a CentOS base won’t suddenly cause application breakage because there is no Debian or CentOS base :-)

Why use SSL/TLS on websites?

Tue, 31 May 2016 07:31:26 -0400

Building a secure web application has multiple layers to it. In previous posts I’ve spoken about some design concepts relating to building a secure container for your app, and hinted that some of the same concepts could be used for building VMs as well.

You also need to build secure apps. OWASP is a great way to help get started on that. I’m not going to spend much time on this blog talking about application builds beyond some generics because I’m not really a webdev. When I wrote Java we didn’t have modern IDEs, tree structures, “factories” and the like, and I never kept up with that. Other people can go on about specifics a lot better than me ;-)

However a strong OS base and a strong app doesn’t mean much if your web server is weak. An important step is using ~~SSL~~ TLS for your communication channel.

Why use TLS at all? Isn’t this just paranoia? Unfortunately not; if you are on a mobile network or using a WiFi hotspot then the network, itself, may insert adverts into the browser; Verizon have added tracking cookies on their mobile network.

By using TLS you are helping to prevent this, and so give your users the site you want them to see, untampered by third party injected adverts. It also helps protect your users from malware that has hit major Ad networks.

Now, obviously, any login that takes credentials (username/password/whatever) should be over TLS to stop the password being passed in plain text over the network. But what isn’t necessarily so obvious to people is that the form that people type into should also be TLS enabled. Why? Because otherwise an attacker could modify the form; as far as the user is concerned they’re entering the data into their normal login screen but the bad guy now takes a copy and then logs them in, as normal. Oops!

What’s even harder for some people to grasp is that any protected page should also be TLS enabled. A typical login process may require you to enter your details into a form, submit these details. If you’ve entered the right data then the server will send back a cookie. This cookie is a token that the rest of the site actually uses to verify you have an authenticated session. If the site is over HTTP then this cookie is passed in plain text and can be stolen. This is a form of session hijacking.

So we have some important use cases:

Prevent network advert injection (and risking malware injection)
Create secure login pages
Login submission pages
Any protected page

Between them all this should be screaming “use TLS everywhere!”. Every page should be via TLS and not by HTTP.

With the advent of Lets Encrypt, allowing for free automatable certificate granting, even a lowly blog like this can be TLS enabled.

This blog is served over https for these reasons; there’s no confidential information, no passwords, no secure data… but I don’t want my content hijacked by network providers. I don’t want my (counts on fingers) 5 readers to risk malware infection because they wanted to read my stuff.

There are some “gotchas” with TLS (make sure you’ve disabled older protocols, weak ciphers, ensure users can’t be downgraded to non-TLS sites, etc) which I’ll get into in a later post. That’ll be more “hands on” and describe how to configure Apache in order to get an A+ SSL rating.

How public cloud can change your security stance

Mon, 23 May 2016 10:20:32 -0400

The core problem with a public cloud is “untrusted infrastructure”. We could get a VM from Amazon; that’s easy. What now? The hypervisor isn’t trusted (non company staff access it and could use this to bypass OS controls). The storage isn’t trusted (non company staff could access it). The network isn’t trusted (non company…).

So could we store Personal Identifying Information in the cloud? Could a bank store your account data in a public cloud?

Could we even store public data in the cloud? Imagine if $MEGACORP stored their TV promotional videos on Amazon; they become the official promotional URLs. And then some replaces it with some porn. Umm…

So there are different security concerns in operating in a public cloud environment and different mitigation strategies required (some of which may be contractual and with $100mm indemnity clauses; some of which may be technical; some of which may just be restrictions on what we allow to run there).

And then once we’ve got the “cloud management” aspect under control, at the end of the day we still just have a VM. Many companies are already pretty good at building VMs. Automation tools can do this in minutes. But then we have the megaton of layered stuff that takes a month of manual work. It needs to hook into the authorization tools, the inventory management tools, the security scanning tools, the change mangement tools…

You were planning on those integrations, right? You weren’t just going to run a production workload on uncontrolled machines. Heck, you weren’t going to run a DEV workload on a machine that can tunnel through your security perimeter…

Building servers is easy. Controlling servers is hard.

So using Amazon just for “replacement IaaS” (Infrastructure as a service) doesn’t really buy us much. Theoretically many large companies can create and run VMs cheaper than Amazon, especially those servers that run 24x7.

Now Amazon IaaS is great for “short running high compute” apps. Let’s say we have a monthly reconcile process that takes 80 physical machines and runs for 24 hours a month. That’d be cheaper to run in Amazon; we don’t need to pay for 30 days of server we’re not using; just for the one day. Neat! But for a 24x7 server that’s always up? We can (and should!) do cheaper. (Hint: many internal service costs includes layered tools and monitoring and network and stuff; we’d have to pay additional licensing fees and network usage charges and so on, on top of the base Amazon EC2 charge).

But this “run a server only when we need it” process requires a change in how we deploy things. This ‘month of manual work’ just won’t do. Existing monitoring, control and inventory management systems are all based around persistent consistent data (your server is flagged red because your license scanning tool can never reach the server because the VM isn’t running!) So now our existing security and audit and access management tools need to be reconsidered.

How does change management and patching work in a world where the server isn’t ever up? So now our vulnerability management processes need to be reconsidered.

Solutions we’ve built for traditional datacenters, based around relatively static compute (inventory changes take place over days, rather than minutes) may not be suitable for a cloud environment.

Then we have regulators; their requirements aren’t necessarily cloud friendly either. Can we meet regulatory controls and demands in this world?

To me, this says “lift and shift” of existing apps from a corporate datacenter into a cloud environment may not be the best options. But if we spend the time and effort… if we build out the datastructures and controls; if we automate application deployment; if we move away from “hands on keyboard” for operate staff (“No, do not ssh into the server; have tools and processes to collect data upfront and then analyse the data, fix the problem and then redeploy the fixed version”). Basically streamline the development process via automated testing/deployment tools (CI/CD FTW)…

Now we can change our security posture. Do we need central authentication in a world where no one can login? Do we need inventory license scanning in a world where we state what apps are running where? Do we need application and config discovery when server configuration is defined programmatically and never changes?

Not all of these changes are dependent on the public cloud. Some of these things we should also drive in the traditional compute environment. Think how much more productive your code release cycle would be if you had full end-to-end automation processes? Think how much more secure your apps would be if no one can login. You might gain many of the perceived benefits of a public cloud internally just by adopting the same tooling.

These operational changes then change our security stance, simply by taking people out of the equation. We don’t discover the state of the world, we define it and automation ensures it.

Why do we need a detail keystroke log of change activity if no one logs in to do changes? Why do we need a central authentication system when no one can login?

The controls move out to the code management environment. How does code get passed through dev/test/qa/prod? What automated scanning tools are deployed? Do you use a CI/CD process? Can you use test-driven processes (check code in, it autobuilds and deploys all tests)?

This changes how companies report their security to external auditors and regulators. It changes how we deploy security technology. And it can actually reduce “time to market” delays!

The risks of Single Sign On

Mon, 16 May 2016 17:22:14 -0400

If your organisation is anything typical then you have multiple web sites and application that require authentication. If you’re lucky then you might have something like CA Siteminder, but your staff still complain about needing to re-authenticate every so often. The more times they need to login, the greater the chance of a mistake, causing a lockout and driving people to distraction.

So you hatch a plan; let’s do a true Single Sign On. And, hey, everyone logs into their Windows desktop and Active Directory presents as a Kerberos source, so why not use AD backed Kerberos as your SSO solution? Now users don’t need to retype passwords all the time.

Problem solved, right?

I feel this approach is misguided, and may open you to greater risk.

The problem is one of “security boundaries”.

In my view staff have at least 3 roles:

Basic “employee” role shared by pretty much everyone; ability to enter the building, login to desktop, read email, enter timesheets… that sort of thing
Stuff done on their own behalf. eg setting up direct deposit of wages, signing up for the company gym. Here the staff are effectively customers of the company, using HR and similar systems
Stuff done specific to their job; developers, SAs, DBAs, bank teller, trader, sales representative, customer support…

The question, in my mind, is whether the gym web site should use the same authentication token as gaining SA shell access to a production server… especially when that access is mediated by a Windows desktop, generally a weak point in any security setup.

Now I’m of the general opinion that any type of Unix shell access on a production server should be considered “privileged access”. Similarly interactive SQL access on databases. Pretty much any access that’s not “end user”. So SAs, DBAs, Application Operate staff et al should not have “SSO” access to their infrastructure. Use something like two factor authentication, or a break-glass process, or some other mediated access solution that requires “step up” authentication. Your desktop AD Kerberos ticket should not grant you access.

This means we can reframe the question in terms access levels:

Basic access (desktop, email, HR-like, etc)
Application access (job specific)
Infrastructure access (privileged)

Privileged access should definitely not be SSO. Application access is somewhere borderline between allowing your desktop credentials to be used or not. It may depend on the risk profile of the application.

The second problem is in a span of control. If everything is controlled by a single authentication source (e.g. Active Directory) then a breach in that environment can be used to pivot across your whole estate. If you have a small number of servers then this risk may be reasonable, but if you have large numbers of Windows and Unix servers then consideration should be paid towards segregating the controls environments. Now if Windows is broken into your Unix environment isn’t directly at risk; similarly if Unix is broken into then your Windows estate is protected.

Of course, more control points means more attackable servers, increased complexity around access admin, auditing, log analysis. You don’t get something for nothing.

It’s a balancing act; desktop, windows servers, unix servers, databases… where do you decide an environment is big enough to warrant its own AAA infrastructure, and how much risk are you willing to accept.

Summary

There has been a drive to merge all authentication systems together; allow Unix, Windows, databases, applications to share the same datasource (primarily Active Directory, via Kerberos). Tools such as Centrify and BeyondTrust Powerbroker (PBUL) allow Unix machines to participate in Active Directory a lot more closely than just using native Kerberos configuration, providing a form of central administration and control.

But you need to consider whether this surface simplicity is the right thing to do from a risk and controls perspective. SSO may not be the correct behaviour, and create a risk concentration around the AD infrastructure.

Ideally, of course, no one should have access to infrastructure and everything is automated :-)

The People Problem

Sun, 08 May 2016 20:48:51 -0400

“To summarise the summary of the summary; people are a problem” - Douglas Adams, The Restaurant At The End Of The Universe

In a traditional compute environment we may have a lot of controls. There may be a lot of audit regulations. Organisations create a lot of processes and procedures. Want to login to a Unix machine? Better have an approved account, with the right authorisations. DMZ machines may require 2FA. Want to become root? Off to a password vault for break-glass (and in some environments, such as Monetary Authority of Singapore, then keystroke logging may be involved!) with management signing off on the activity afterwards.

That’s a lot of effort needed, just to login to a machine and make a change.

Changes… I hope you had a approved change record for that activity. Do we need tripwire to verify only the right files were touched? How can we verify you didn’t read “supersecret.txt” while you had root access?

So let’s not do that. Let’s not login at all. If you can’t login then you don’t need all of that technology and processes, and all the fun that involves. Script everything. Automate everything. Hands-off deployment.

“If you have to SSH into your servers, then your automation has failed” - Rich Adams, https://wblinks.com/notes/aws-tips-i-wish-id-known-before-i-started/

This is how we need to manage cloudy environment. We should automate pretty much everything. Ideally our physical servers can not be directly reached. From bare metal we should build our servers, our network, our storage, our management plane and the complete cloud environment via automation, in a few hours. No human login needed.

This automation also allows us to change the way we operate. We never patch a machine; we rebuild it from scratch. The automation brings in the latest version of the packages and the fresh deployment is our fully patched server. If a problem such as heartbleed or shellshock occurred again we are in a position to redeploy all of the servers in a rolling upgrade in hours.

This is also the only way to scale. If you’re building a dynamic elastic environment (if you’re not, then why are you using cloud?) then you may need to build 1000 servers that run for 2 hours then destroy them again. How will you do that without automation?

This is just the next step in a trend. When I started doing stuff 30 years ago, servers had a shared well known root password. DBAs knew this password. Some “trusted” developers knew this password. SAs would login as root and stay logged in for weeks at a time. Slowly we removed access starting with “root”; the developers screamed that they couldn’t deploy their programs because they needed root to do it; the DBAs screamed that they couldn’t extend their databases because they needed root to do it; the SAs screamed that they couldn’t do… pretty much anything, because they are root. And now we’re at the stage where no one even has a login!

And if you can’t login, you can’t break it so easily. We’ve also removed a massive threat vector (no server login means the accounts can’t be brute forced) and so reduced the risk.

By removing human access from the servers we have simplified the environment and made it easier to secure.

And now we’ve got this concept, why stop there? Let’s retro-fit these controls into the traditional compute environment as well. Automate application delivery; automate patching; automate… everything!

Shadow IT

Mon, 02 May 2016 11:15:23 -0400

Shadow IT isn’t a new thing. Any large corporation has seen it. Sometimes called “server under desk” or “production desktop”.

Sometimes it grows out of a personal project that started on a spare machine and that gradually morphed into a mission critical machine… but without any of the controls and tools normally associated with production infrastructure (patches, backups, DR, access admin, security scanning…).

Other times it grows out of a desire to do things quickly; all of those controls and tools take time and can hinder the developer experience. Let’s hack together something now and then migrate it later… except the migration never happens and that under-desk server stays forever.

We know this is bad but it still happens. Central IT is frequently seen as more of an inhibitor than a facilitator

Despite this, at least this shadow IT was within the corporate boundary; there was some level of protection provided by border firewalls.

The new Shadow IT

These days you don’t even need to rummage up an old desktop to turn it into a server. Just wave your corporate card at Amazon and, just like magic, you have a server. Two servers. Ten servers.

Now the situation is a lot worse. Not only are these unmanaged and uncontrolled servers, but they are also outside of your security perimeter. On the internet, facing attacks. Is your application secure against hacking? Do you have strong passwords? Did you turn off all unnecessary services? How long will it be before someone brute-forces your root account?

The result: data exposure. The personal data of 93 million Mexicans were exposed because of just this type of problem.

The first you may know about it is when your CEO is being contacted by the Wall Street Journal for a quote, or when Troy Hunt adds the data to Have I Been Pwned.

Summary

Shadow IT is pernicious. It is hard to stop. Instead we need to lessen the incentive to do it; make it easier to obtain the resources in a controlled manner.

This is a culture change. Abstract away from the Operating System and focus on what the developers need; a PaaS, obtainable by self-service processes; ability to push code into dev and automate promotion to production; automated scanning, backup, resilliency… take all this off their plate and provide it at the click of a button.

Now there’s no need to wave your credit card at Amazon!

Vulnerability, Threat, Risk

Wed, 27 Apr 2016 09:54:33 -0400

From Twitter came this gem:

This is a cute way of helping people understand the difference between the three concepts. It also helps start to drive conversation around remediation activities and risk assessment.

(Let’s not get too tied down with interpretation; all analogies have holes :-))

What if the door was a bedroom door, rather than a house front door? How does this change the probability of a bear getting in and thus getting mauled? Or does the open bedroom door cause a new threat (room mate seeing you in a position you would find embarrassing)?

Not every risk is identical. An open door leading to a balcony that’s 30 floors up is unlikely to have a bear come through it! If your exposure to the threat is minimal then the risk of exploit is minimal and so your remediation timelines need not be immediate.

When we look at security we tend to have an “attackers will get in” mentality. “Close the door because the bear will get in”. It’s a fair position to take, especially since attackers can chain vulnerabilities together (the front door was open and the bedroom door was open).

So we take a security in depth view; don’t trust a single control (front door was closed, but the kitchen back door was open) but layer controls. More prosaically, don’t assume that firewalls will keep out attackers; there are other ways of getting remote code execution (email client bugs, phishing, desktop vulnerabilities, bad code… or even a simple misconfigured firewall!)

This “in depth” view should use different types of controls that overlap. So you wouldn’t run two different virus scanning tools on your desktop; instead you may run a virus scanner and a firewall (to block outgoing connections to untrusted sites) and a filtering proxy (for trusted outgoing connections) and…; controls that may overlap (the proxy may do virus scanning as well) but do other stuff; if you do get infected then the code can’t reach out to the IRC control channel because of the firewall; it can’t download more payload because of the proxy.

Conclusion

When you identify a vulnerability you should evaluate the threat and the risk to your environment. What’s the worst case scenario of the vulnerability is exploited? What’s the chance that it will be exploited? What mitigations do you have in place to prevent the attack vector from being exploited?

Remember, though, that sometimes you have invited the bear in (insider threat from disgruntled employees). That needs to form part of your risk assessment!

Container technology

Mon, 25 Apr 2016 15:43:01 -0400

I’ve spent a few posts talking about the ecosystem required to keep a container secure; hands off automation, code provenance, and the like.

But a number of people have asked me about the techology. Mostly they talk about “docker” and the security concerns. I’ve been loathe to talk about technology specifically because it changes. Yesterday docker daemon runs as root; tomorrow it may not. Yesterday the kernel exposed a problem, tomorrow it won’t. I think that by focusing on the technology implementation you’re missing the bigger picture; the management of the ecosystem is more important than the technology itself.

But with this in mind, here’s a few thoughts.

Containers share a kernel. The container technology tries to isolate each instance from each other. This means that all the interfaces exposed by the kernel to the code form part of the attack surface. There are over 300 system calls. Each of them need to ensure the isolation is maintained. And any new syscall added needs to maintain the isolation.

Containers make use of various kernel technologies to try and minimise the risk. Obvious techniques include having a separate network interface that’s only visible inside the container (let’s hope there’s no data leak between packets on different interfaces), having a separate view of the filesystem (chroot on steroids), a separate view of the process space and so on.

In addition, with latter kernels, user and group ids can be mapped inside containers. User zero inside a container need not match user zero outside of it.

Then we can start to look at AppArmour and SELinux to add to the security. Mandatory Access Controls (MAC). If you are rolling your own solution then you might also want to look at something like Computer Associates “Control Minder” (previous known as eTrust Access Control, or even SEOS; it’s changed names a lot). This is a highly flexible MAC system; a lot more flexible than SELinux. But not cheap :-)

Now a clever one; seccomp. This allows you to drop permissions to use specific syscalls. If you can’t use them then you can’t leverage any isolation bugs; basically reduce the attack surface area.

Use of CAPabilities inside the container may allow some actions to be done without needing root (eg “ping”). Ensure you drop unwanted capabilities before starting your app.

Of course these are all kernel oriented, and so themselves form part of the attack surface. Older Android machines have had kernel level exploits due to SELinux bugs!

You can see that there’s a lot to juggle and keep in mind. It’s a full time job. Unless you have a specific requirement to build your solution, I’d avoid it and use something more “off the shelf”. After all, how many people build their own OS?

By using something common (Docker, Rocket, Cloud Foundry, etc) you’ll have the advantage of a lot of developers, constantly improving the product. Docker 1.10 makes use of many of the tools listed above; it works with seccomp, SELinux, AppArmour. It deals with name spaces…

Conclusion

Don’t get too hung up with the technology implementation. Be aware of what you’re using (e.g. docker daemon runs as root) and the best practices around it. There’s hundreds of pages on the web about that!

Pick a technology stack that meets your use case (Docker? Rocket? Mesosphere? Cloud Foundry? Google Function? Amazon Lambda?) and maintain it like you’d maintain any vendor product.

You’ll gain a bigger win by focusing on your processes and automation. Which you need to do, anyway, to ensure the content is secure. And the nice part of that is by fixing this part you can scale your solution from micro-service containers all the way up to VMs or even physical server builds. Hands-off automation all the way!

Business cards

Fri, 22 Apr 2016 10:31:19 -0400

A decade or so back, VistaPrint did a “free card” offer as long as you used one of their templates. So I got a bunch of cards printed

Over the years I’ve probably given out…5 of them? Heh.

VistaPrint no longer seem to do freebies, but I decided to refresh my image.

The cost was $8 for 150 cards or $9 for 250, so I went for 250. And then after checkout they said for $1.50 more I could get another 250. So total of $10.50 gets me 500 cards.

I expect they’ll last until after I’m dead!

But if I’m gonna start going to conferences or give talks (maybe!!) then having a card might be useful.

Maybe containers are VMs after all

Thu, 21 Apr 2016 22:06:42 -0400

Back in Container security I said that we need to think about containers as VMs. I then looked at an easier way of looking at containers, by not treating them as VMs. Hopefully, at this point, some of you were thinking “Hmmm!”.

Finally I discussed the processes and workflows outside of the container implementation that is needed to keep containers safe (build processes, etc).

We can turn what we’ve learned on its head.

In order to build and maintain a container ecosystem we need to think of containers as:

Transient
- There is no persistent storage
Short lived
- Anywhere from microseconds upwards
Immutable
- You don’t change the running container; you build a new image and deploy that
Untouchable
- You NEVER LOG IN!
Automated
- If you can’t login, you better automate the builds

We need processes around the maintenance of containers:

Build good containers
Scan existing containers
Replace bad containers

Why should this apply solely to containers?

Let’s look at an Amazon EC2 instance. Ideally we want to spin these up “on demand”.

We shouldn’t store stuff in the VM disk image; either use Elastic Block or S3 store or a database. For elastic compute where the same data needs to be shared, S3 or a database. Basically the data is transient.

These EC2 instances aren’t necessarily short lived; they may live for hours or days… but, really, compared to traditional compute environments this is short lived.

We could design workflows so that AMI images are not changed; new ones are built. Don’t patch in place, reboot from patched images.

And, of course, never log in. You automate your processes. Your developer teams instrument their apps.

The key to all this is proper automation.

That all sounds like what I defined as a container characteristic!

Summary

So here’s the challenge; maybe for the past decade we’ve been managing VMs wrong! We have old heavy-weight processes. This speaks somewhat to the pets vs cattle discussion, but coming at it from the other direction.

Maybe we can treat containers the same as VMs. Or, more accurately, we can treat VMs as if they were containers.

Unix Identity and Access Management

Sat, 16 Apr 2016 19:41:39 -0400

Identity and Access Management (IAM) historically consists of the three A’s

Authentication
- What acccount is being accessed?
Authorization
- Is this account allowed access to this machine?
Access Control
- What resources are you allowed to use?

On top of this we may also need to consider

Auditing
- Log the attempt to use the machine
Provisioning
- How does the account get onto the machine?

“Access Control” is not directly under control of IAM. In the unix world this is defined by group permissions on files (group membership is part of IAM), ACLs and so on. For this blog entry I’m going to ignore it… mostly.

Let’s call the result (excluding Access Control) AAAP.

Historical Unix

In Unix we started pretty simple. We had the password file /etc/passwd. The second field of this was a one-way encryption using your password as the key. The AAAP answers are now simple:

Authentication
- Did you enter the right password?
Authorization
- Is the account in the passwd file?
Auditing
- Log to syslog (and maybe elsewhere)
Provisioning
- A superuser added the account.

(This password file also defined your primary group. Another file /etc/group defined your secondary groups; this is where IAM impacts Access Control).

In this world the mere existence of the account was authorization.

Aside: this process still works today in an “out of the box” Linux build. The crypt string has moved into /etc/shadow because the old DES algorithm was eventually too weak and open to simple attacks; a modern machine will use SHA2 instead of DES. But a machine can still be setup so it works just like it did 30 years ago; add the user to /etc/passwd and /etc/shadow and they can login!

Yellow PagesNIS

This started to hit scaling problems. It was OK if you had 1 or 2 machines, but 10? 20? 100? Many organizations started to write simple synchronisation scripts to try and keep the password in sync but these didn’t scale too well.

In the ‘80s Sun Microsystems invented a solution called “Yellow Pages”. Eventually (due to British Telecom trademark issues) it got renamed NIS (Network Information Services). Now what NIS did was to create a network version of the password file (and other files, out of scope for this discussion). You can add a user to the central map and it will show up on every machine. Scaling solved!

A new problem showed up; you may not want every user to login to every machine. So NIS also included the concept of a “netgroup”. You could create a netgroup called “sa_team” and put the SAs into it. Another netgroup could be called “dba” and your Oracle DBAs go into that. Developers could go into “dev”.

Now your passwd file consists of users, as before, and @netgroup entries. So every machine would have @sa_team but only the Oracle servers would have @dba. AAAP now reads:

Authentication
- Did you enter the right password?
Authorization
- Is the account in the passwd file or in an included netgroup?
Auditing
- Log to syslog (and maybe elsewhere)
Provisioning
- A superuser added the account or a netgroup.

There’s still a scaling issue (every machine needs to be configured for the right netgroups) but this is normally a one-off event or easily scripted event. You don’t need to touch hundreds of servers every time someone leaves or joins.

As a technology NIS had a number of security issues (e.g. spoofable, allowed the crypt strings to be seen) but the ideas worked well. Similar ideas still persist today, using LDAP instead of NIS. (We’ll not talk about NIS+ and pretend it never existed).

PAM and NSS

What more do you need?

Flexibility.

Until now every program needed to know about all the backends. The programs like login/ftp/telnet/rsh on a SunOS machine all knew about NIS. But that’s all they knew. You couldn’t add a new backend without replacing every “front door” program.

Starting in Solaris 2.5.1 and made better in Solaris 2.6, Sun created a new structure “PAM” - Pluggable Authentication Modules. Modern Linux also has PAM. Even AIX now has PAM (after going down its own path for a while).

PAM and its partner NSS (Name Service Switch) are (at first glance) a complicated process. They splits out something that was done by the passwd file into two separate sections:

Naming Services
Authentication/Authorization

Naming services are handling things like “what UID is login fred?” “What account is uid 12345?“. NSS allows for different backends (the local passwd file; a NIS server; an LDAP server; a MySQL database…). It takes what Sun created in the early NIS days and improves it to an API so every program doesn’t need to know how to talk to every backend; they call (via libc) a common stack. This turns out to be very powerful and persists today.

PAM is where things get more interesting. Historically we’ve seen that “authorization” was simply based on a simple rule such as “are you in the passwd file or an included netgroup”. PAM potentially allows any rule to be defined; “Harry can login from these machines between 9am and 5pm”; “Fred must use an RSA token to login”. Normally you wouldn’t go to these extremes (you’d use “role based” rules - I’ll cover this in a later blog) but because the rules are now programmable you can do everything.

It also opens up a difference between traditional unix and modern unix. Previously if you were in the password file then you were authorized. “id fred” was enough to tell you if you had an account.

Now PAM may deny you access even if you exist. “id fred” may say that Fred has a login, but PAM may still deny him.

AAAP gets more complicated:

Authentication
- Does NSS say your account exist? Do the pam “auth” rules say the right credentials have been provided?
Authorization
- Do the pam “account” rules permit this login at this time from this machine with this credential?
Auditing
- Log to syslog (and maybe elsewhere)
Provisioning
- Are you in the datastores queried by NSS?

All is not good in paradise

We can now handle lots of things. What could possibly go wrong?

Auditing is traditionally just syslog. Remote syslog is unreliable (UDP packet loss) or insecure (TCP packet sniffing). Authentication logs should be considered high risk; who hasn’t mistakenly entered their password into a login prompt? “Failed login for mypasswd on ttya”; “successful login for fred on ttya”… can you guess Fred’s password?

So audit logging needs to be handled better.

Then there’s the provisioning stage; how are the rules for NSS and PAM setup and maintained?

A common setup is to allow “passwd: files ldap” (or similar). This means “look in the local passwd file first; and if the user isn’t found look in LDAP”. So far so good. But this means that an SA could add “myacct” to the passwd file and login with that.

What happens when that SA leaves? We may delete his LDAP account, but “myacct” is still there”.

A number of organizations with an IAM function (e.g. banks, pharma) have a “separations of duties” control structure. One team controls access (an IAM team) and another machine controls the machine (the SA team). If the SA team can add users then we have a separation violation.

We’ll deal with this problem in a later blog. This is where IAM gets “interesting”.

Summary

Modern Unix systems use PAM and NSS to control access to machines. In their default configuration they look just like a machine from 20 years ago; add user to /etc/passwd and crypt string to /etc/shadow and the user can login (if it wasn’t for /etc/shadow then this would be 30 years ago).

But they are a lot more flexible and can do a lot more than that. Want single signon from your windows desktop to the linux machine (no need to enter your password!)? The PAM kerberos module can help. Want to pull in usernames and groups from your Active Directory tree? NSS is the answer.

PAM and NSS allow you to write code that’ll do almost anything you want.

Keeping containers safe

Sun, 10 Apr 2016 10:18:21 -0400

In a previous post I showed that if you stop treating containers as if they were VMs then container security is easy.

Now we need to look at how to keep the contents of containers safe.

In general there are a number steps:

Build good containers
Scan existing containers
Replace bad containers

Build good containers

This should just be an extension of your existing source control process; your CI/CD process; your “test driven” processes. Whatever you use :-)

A container won’t magically make an application secure, but if you build them with “known good” code and only put things inside the container that need to be there then you’ve made a good start.

Some guidelines, which should just be common sense good coding practices anyway. These guidelines apply to traditional compute environments on physical machines, on VMs, on containers. Control your builds!

Don’t create a whole copy of the OS to put inside the container

Some PaaS systems (eg Cloud Foundry) will provide the necessary infra via “stemcells”. If you’re using docker then you might want to use a baseline to layer on top of. If you’re building a container from scratch then only put in code that is essential. If code isn’t present then it can’t be exploited. Minimise the footprint; minimise the attack surface.

Don’t pull code direct from the internet.

In March we saw how one person removing code from npm caused build systems from all over the internet to fail. Your builds must be repeatable, and in order to do that you must control the dependencies.

Another risk of pulling from external repos is that if you don’t define your dependencies to the specific version then new code may be introduced upstream and break your app. Or, worse, rogue code gets introduced that creates a backdoor.

Similarly if you use a binary repo (eg docker) then don’t have build time dependencies on them; bring the objects in-house.

This may mean you have your own internal repo and code against that. You’ll also need to maintain this repo, but at least you get the chance to scan the code, verify the licensing agreements, etc. Do you want your code to be BSD licensed? Better verify none of the code you’re using is GPL’d.

Code scan

Everything you build should be scanned. Binary objects should be scanned. Results should be compared to known CVEs and to good coding practices (eg those from OWASP). Good tools can identify bad coding practices that could lead to exploits.

Don’t rely on these, of course. You should always strive to write good code. But scanning can be a useful backstop to catch mistakes.

Automated deployment

The resulting image should go through your normal test/QA processes and then the passing image pushed into your “registry”. Production containers should only be created from the approved registry, never from ad-hoc code.

Scan existing container images

You’ve just jumped through a hundred hoops to build your good container image. You can relax, right? Nope.

Unfortunately code has bugs in it. I know that’ll come as a shock! That code you scanned earlier; that library you depend on; that OS you based your image on… a new bug has been found in it.

Fortunately you have your registry of good containers. You should keep rescanning these containers. When something like shellshock appears you will be able to identify what images are vulnerable.

The action you take may depend on the risk. Perhaps you can block any new instances from being created from this image. Maybe you’ll just flag it and make the owner responsible for fixing within 90 days. You may be able to automate repair (eg vendor provided an OS level fix; patch the base image, auto-rebuild the dependent images).

What you do here is very specific to your environment and risk exposure, but it starts with knowing there’s a bug to be fixed. That’s what this repetetive scanning does.

Replace bad running containers

It’s all well and good fixing your image registry, but what about all those containers that are running? You need to know where each image is running so that you can shutdown and replace the bad image with the repaired one.

Your “orchestration layer” of the automated deployment should do this for you; as it builds a running container it should track the container (where it was deployed; what image it came from). When the container is destroyed it needs to track that as well. Now, when your image has been corrected you can push out the good version, remove the bad version. You can also report on known bad running images in order to identify your existing risk.

This is very important in a container world simply due to scale. You may have a million containers running around; you can’t keep track of them all… but your orchestration layer can! Or your service discovery layer. Or some other tool. Something has to track your containers.

Summary

Most of this is identical to the processes you should already be following for good code deployments. Containers don’t bring much new to this, although they may change how you do it. Containers also encourage the development of 12 factor applications.

The challenge, of course, is that many enterprises don’t follow these processes. They allow a wild west development style and call it “DevOps” and “agile”; they focus on speed of delivery and sacrifice many of the controls that are necessary.

Take time, now, to build the management tooling and automation of these steps. Then you can genuinely build an environment where your developers can get code live using whether named process they like. Your automation will help encourage good security, good code design, and also remove hindrances to deployment.

Container Security is Easy

Thu, 07 Apr 2016 20:02:40 -0400

People think container security is hard. But it’s not… if you think about it the right way. And that’s where people tend to go wrong, and that’s why they think it is hard.

So let’s follow a thought pattern…

First we need to consider what is a container and what distinguishes it from a virtual machine.

In general a container has the following properties:

Shared kernel
Segmented view of resources
- Separate process ID space
- Separate filesystem mount space
- Separate IPC memory segments
- Separate view of the network
- …
Multi-platform
- Linux VServer (from 2001!)
- OpenVZ
- AIX WPARs
- Solaris ~~Containers~~ Zones (from Solaris 10, 2005)
- Linux containers (LXC, cgroups, etc)
- Docker
- Warden
- …

You can see that containers aren’t even new. I was using VServer on a RedHat 6 build (not RHEL; the old old RedHat!) in 2001 to create “bastion servers” on my firewall machine; one bastion had an apache instance, another handled incoming SMTP, another handled incoming ssh. (Yes, yes; overkill for a home network at the end of a 640/128K DSL connections!)

Solaris tended to treat their containers as lightweight VMs; you could install a whole copy of the OS or you could create ‘sparse zones’ which used the parent OS image as a read-only copy inside the zone. There was a persistent filesystem (typically from ZFS) associated with the zone.

There are plenty of ultra-cheap “virtual machine” providers that use OpenVZ and similar to give you a “server”. It’s not a VM, but it’s mostly good enough.

So what goes into a container?

Almost anything. As we saw from Solaris, a container could contain a whole OS install. Including an ssh daemon, an IP address dedicated to the container and so on (so now we have identity and access management issues; remote scanning issues; change management issues…)

On the other hand a container could contain the minimum necessary to run the code. My “web” bastion server from 2001 had httpd and the minimum necessary libraries and binaries to run it.

With a language such as “go” (which compiles to a static binary) a container could contain as little as a single binary plus a few pseudo devices to make it work. It could have an “internal to the machine” ethernet address with the parent OS port forwarding traffic on a single port.

Over on the docker blog, Mike Coleman argues that containers are not VMs. He’s right that they shouldn’t be. Unfortunately that’s how some people use them. In an earlier post I highlighted how some people may treat a container as VM and flagged that this is wrong.

If containers are so flexible, how do I secure them?

Let’s look at the worst case scenario. A container may have:

A cut down (maybe!) version of the OS
- Glibc, bash, web server, support libraries, tools
- Maybe different versions than those running on the host!
Filesystems
- Private to the container?
- Shared with the host?
Access to the network
- Bridged to the main network?
- NAT?
Processes that run
- Network listeners
- User logins

Despite Mike’s comments, this sounds like a whole operating system to me. The only difference is the shared kernel.

So, in the worst case, we need to control this the same as the OS. And deal with the unique challenges.

It’s easy to draw parallels. To start with:

	Virtual Machine	Container	New threat
"Parent Access"	Hypervisor attack; one VM may be able to use hypervisor bugs to gain access to another VM	Parent OS attack; process may be able to escape the container	More people may have access to the host OS than to the Hypervisor; more threat actors. Shared kernel is bigger than hypervisor; bigger attack surface Kernel may allow dynamically loaded modules; variable attack surface!
"Shared resources"	Rowhammer Noisy neighbour Overallocation ...	Mostly the same	Resource separation is now in the kernel; not as well segregated; not all resources may be segregated
"Unique Code"	Each VM may run a different OS, different patch revisions, different software	Each container may run different software versions, different patch levels, different libraries	Scaling issue; we can run many more containers than VMs
"Inventory management"	What machines are out there? What OS are they running? Are they patched? What services are they running?...	Mostly the same	We typically have strong controls as to who can create new VM's but we allow anyone to spin up a new container 'cos it's so quick and easy!
"Rogue code"	Is the software secure? Acceptable license terms? Vulnerability scanned?	Mostly the same	Anyone can build a container; it may have different versions of core code (eg glibc) than the core OS. Developers may introduce buggy low-level routines without noticing

I think it becomes clear that, in the worst case, there are no new problems. We see all these at the VM level.

So can we use the same solution?

Where are things different?

Unlike a VM, a container may have other characteristics:

Very dynamic
- Harder to track, maintain inventory
- If you don’t know what is out there, how can you patch?
- How many containers may suffer shellshock, heartbleed or CVE-2015-7547 (glibc) issues because we don’t know where they run?
Technology specific issues
- Docker daemon running as root?
- SELinux interaction
- Root inside a container may be able to escape (eg via /sys)
Kernel bugs
- Ensure your parent OS is patched!

What is now highlighted is that the existing solutions may not scale properly. What works for a 1,000 (or even 10,000) VMs won’t work for 100,000 or millions of containers.

Can your solutions deal with containers spinning up, running for a few minutes, then shutting down again? Probably not.

The easy way.

The above is the hard way of doing it. You can manage your containers that way if you want. If you do then you’re just using the container as a lighter-weight VM. You’re likely to hit problems. If you want a VM then build VMs. Automate your VM build processes; don’t rely on docker repositories (I hope no enterprise is using external docker repos, anyway… that should go without saying).

Instead, if you want to make use of containers you must follow Mike’s advice and stop thinking about containers as light-weight VMs.

Now you can start to think of the flexibility containers can give you. You can scale bigger and better.

While container technology allows you to treat a container as VM, you should think of a container as

Transient
- There is no persistent storage
Short lived
- Anywhere from microseconds upwards
Immutable
- You don’t change the running container; you build a new image and deploy that
Untouchable
- You NEVER LOG IN!
Automated
- If you can’t login, you better automate the builds

Now we can start to build a container security model. We have an application focus, rather than an OS focus. You don’t allow people to deploy containers; your “application delivery system” does it for them.

Now we’re on the path to elastic compute; if your app deployment is hands off then you can start to scale!

Minimizing attack surface

We can follow basic hygiene rules:

Harden the kernel
- You don’t have to use the vendor provided one
Ensure the OS is patched
Integrate into your CI/CD process
- No manual startup of containers
- Only those that passed testing, vuln scanning, etc can be run
- No external code without approval
- Standard code hygiene (provenance, licensing, etc)
Automate, automate, automate!

This gives big wins.

How many licenses are you using? You know because your automation tools will tell you what is running
A CVE has been found; you know what images are vulnerable and where they are running
- Your image is more lightweight so may not be vulnerable in the first place!
Patching doesn’t exist; you just redeploy

And another benefit… if your app can be delivered this way, you can also start to deploy VMs this way. A Linux OS without any form of login? Let’s see someone brute-force ssh bad passwords that way :-)

But remember

Container security is weaker than VM security; the kernel has a large surface area to be attacked.

VM security is weaker than physical machine security.

Network security is weaker than air-gap separation.

Air-gapped machines are weaker than a server covered in cement and dropped into the bottom of the ocean.

We’ve seen that using a container as a light-weight VM doesn’t make sense; you haven’t solved any security issues, you’ve just increased them!

Your security stance depends on your needs and risk evaluation. Pick the technology that is right for you and use it appropriately.

Summary

Google have been using containers for around a decade. Docker has made massive strides in bring containers down a easily automatable format.

But containers ~~are not~~ should not be VMs.

Google Functions? Amazon Lamba? They’re containers! A rules engine triggers on an event, starts up a container, runs your function, destroys the container. A container that spins up, runs, is destroyed in a fraction of a second. How cool is that?

A PaaS like Cloud Foundry; your app runs inside a container. The autoscaler will spin up new containers across the cloud. A VM dies, the PaaS will spin up a new copy. All you deal with is your app code; you don’t touch the OS.

Docker can build light weight containers that can then be thrown into Mesosphere; a Swarm of apps can be started up in containers across the cloud quicker than you can read this post.

And they all work by following the basic rules

Transient
Immutable
Untouchable
Shortlived
Automated

I think this post is long enough; we still need to touch on lifecycle managment to complete the security model.

But you already have that… right?

Container security

Sat, 02 Apr 2016 20:29:33 -0500

It started with a set of slides by a friend:

My first thought was to wonder wonder how heartbleed, shellshock, cve-2015-7547 and the like fit into this story. He answered “rebuild the world and redeploy”. Which I felt missed the problem.

You also need a level of control around what goes into containers, who can build containers, where they get deployed.

We have decades of history of knowing that self-run machines are badly patched and badly maintained; if the bug isn’t in the application code then it’s mostly invisible to the developer. Hell, even SAs are bad at patching unless forced (“my server has 1000 days uptime!” is not a good brag).

What has changed? Why will we expect developers to start maintaining their private OS instance (which is, effectively, what a container is) now? Worse, in an environment where everyone and their dog can spin up containers then you may not even know where your vulnerable systems are.

With an opinionated PaaS like Cloud Foundry the developer gets no say; it’s handled under the covers. But if you open up containers without control?

The story I hear (and Adam’s slides seem to go in that direction) is that containers are great for self-service; allow the developers to push whatever the hell they like; they have the ability to fix it, to make sure the code works; they are now masters of their own fate and they can’t break the host.

But nowhere do I hear “security”. Indeed Adam’s slides specifically permit insecurities:

    No need to worry about doing it the right way: just throw the
    library binaries, framework templates, ancient, buggy,
    security-vulnerable versions of Java, whatever in there
    higgedly-piggledy!

This is a bad view of security. If you’re running buggy code (“I’m looking at you, Wordpress!“) then your container gets pwned. OK… not the whole host, but there’s now an RCE in your environment. And it can talk to the persistent data stores. And can exfiltrate that data. And can be used to pivot to other internal services that may not have been exposed… and let’s hope the parent host has a patched kernel so it doesn’t allow container escape.

From a security perspective we need to look at containers as if they are a VM of their own. They have the same vulnerability footprint as a real OS, plus new additional ones. We can’t use traditional external-scans to help detect faults; they were never a good idea, and simply won’t scale to a dynamic elastic environment. We need to build this in.

That’s not stopping DevOps; we need to embed the culture of security into the development pipeline. Have your Jenkins job do a vuln scan during your auto-build. Have a deployment engine that tracks what is where. Have a control mechanism to hard-stop pwned services.

Sorry; despite the lure of freedom, containers must not become the wild west; we’ll lose, big time, if we let it.

New site

Sat, 02 Apr 2016 07:55:25 -0400

My old site was nicely hand crafted HTML. Each bit loving created. It worked… but it did smell a little 90’s. Which doesn’t surprise me; the last time I did any web development was the 90s!

So I thought I’d try something a little more modern.

Unfortunately most CMS systems (eg WordPress, Joomla, Drupal) appear to want to use a database of some form. The content is displayed dynamically based on the user request and the database content.

I have 3 servers (a Linode, a Panix v-colo, and an OVH So You Start colo server). Each one has identical content and could stand in as a DR for the others for web, mail, DNS.

Using a database backed CMS introduces challenges (eg database replication, backups) as well as potential performance issues.

I wanted something that was more “static file”.

Some net-friends pointed me at some site generators. Some were written in python, some in Ruby, some in perl. They have large run-time dependencies to build the site (none for actually serving it).

But Hugo is written in Go and so compiles to a static binary. Ooh! This looks interesting…

I played around with some themes and decided on the Blackburn theme, with a few tweaks.

Hugo has some nice features, including a built in server. I can write a new markdown document and immediately Hugo will detect this and publish it. It’ll even cause the viewing browser to refresh. So in one window I can type and :w (yes; vi) and the browser in a background window automatically refreshes. That’s cool!

Once I’m happy with the content I can then tell it to generate the static content. And then I can rsync it to my 3 servers to publish it.

The one downside to this theme is that it does call a number of off-platform javascript and fonts. I’ve tried to modify it to host as much locally as possible (from a security perspective, calling random javascript from a random CDN seems wrong) but there’s still a large amount. Sigh.

I also debated having a comment system. I’m of two minds. Currently I have Disqus set up. This does allow anonymous posts and does a level of spam trapping for me. I’m not a total fan of Disqus (tonnes of javascript, cross-site tracking) and may turn it off. But it works.

My ultimate aim is to start blogging more frequently (hence the blog format of this new site). Mostly writing about security stuff, probably. I’ve made my content CC BY-NC-ND, just in case I write something someone else likes :-)

I’ll probably keep the old site running because some stuff (eg reference information) isn’t too suited for a blog format. We’ll see.

About me

Fri, 01 Apr 2016 12:26:31 -0400

I first got into Unix in 1987. SunOS 3 on the Oxford “mclab” network (who else remembers “NFS server pampa not responding, still trying”?). At the same time I had access to the “High Level Hardware Orion” machines, in the Clarendon Nuclear Physics lab as a member of Oxford Microsoc.

I had no idea what I was doing. But I learned pretty quickly!

My first job was as a SA for Papachristidis Limited; a Greek shipping based out of London. It was a small office so I ended up doing almost everything (coding, networking, SA, fax machine, telephones, even the door entry security system). It was a good place to learn and hone my skills.

At this point I started with Linux (0.11 kernel) and even got interviewed by Linux Journal.

Around the same time I took over the running of Spuddy (link to old old old spuddy website), which provided free email and usenet in the UK. Basically a Sun 2 with a Magma serial port extender and a handful of modems. A British Telecom engineer once claimed I had more phone lines in my spare bedroom than some companies! Spuddy even got written up in the national computer press and was compared favourably to commercial providers.

My next job was at VNU Business Publications in London, working in a startup division called MatriX Publishing Network. (I remember one of the staff members claiming “We got Spuddy!” when I accepted the contract). I did SA, networking, web development, I even wrote a few magazine articles. The division got shut down (we were making a profit, but not in the core business areas) and I moved into the central IT team where I began my path of automation and centralisation; rolled out Big Brother, created a paging sytem, backups… Eventually I had the Unix servers under control so they made me learn Oracle DBA… then Windows NT4 admin… and then Lotus Notes (at which point I ran away).

In 1999 I started to work for JPMorgan (now JPMorganChase). I’ve been in various roles in that company. Most recently I did cloud security engineering. And in 2016 I started with First Data.

DISCLAIMER: Nothing I write can be considered official policy of any company I have worked for. It’s just my opinion. And, hey, I have opinions! In the long long distant past of usenet I had a signature:

      The truth is the truth, and opinion just opinion.  But what is what?
       My employer pays to ignore my opinions; you get to do it for free.

I hope my employers now listen to my opinions, but you can still ignore them!

But my life with computers started before then. In 1981 I started to play with the school’s Commodore PETs. I taught myself 6502 assembler from a book and hand assembled some simple stuff. Like lots of kids I typed in listings from magazines and then debugged the typos.

For Christmas 1983 I got my own computer; a BBC Micro. I still have it!

So 1984 was when I started to hit my stride. And that’s when I first learned that people will break things; without trying! My brother was a great help, here. I recall writing him a simple “football score emulator” (each team had a random weighting and the program would create score cards, similar to that shown on TV every Saturday). I presented a menu and expected people to enter “1-22”. My brother typed in the team name. Whuh? Naturally the program crashed (out of range error).

And this is where I started my “security” journey. All the way back over 30 years ago. I taught myself defensive coding; sanitise and validation of input. As time went on those techiques grew into general purpose rules (e.g. “don’t trust the client”).

When the web became popular and people stuck validation routines into their javascript but didn’t implement them at the backend then I laughed; and demonstrated their folly.

But I’ve never been a hacker; not really. I’m not a “security researcher”. I’m not a trained security individual (no CISSP, here!). I don’t play “defense” or “offense” in the infosec world, although I have a tendancy towards defense.

What I am, though, is a security aware person. I have over 30 years of experience and can apply it to new situations. Build “secure from the start” solutions. Take a newly found bug and evaluate it. This puts me somewhere between “risk evaluation” and “security”. It’s a role I don’t see many people filling!

Which leads, inevitably, to my CV (or resume, in American) (PDF).

Note that this blog is inherently an “enterprise” view of the world. An organisation with controls requirements, potentially regulators and auditors.

But some of the concerns also affect everyone; do you want to run code that is hackable? That causes an RCE (Remote Code Execution) and so people can “pivot” and attack other servers? Even education research environments should care about this.

So if you’re interested in a non-sensational (I hope!) viewpoint on modern internet security as it pertains to the enterprise, maybe this blog will be a good place to look.

I can’t promise frequent updates; I can’t promise 100% accurate evaluations. But it may still be interesting.

If you need to contact me for any reason and leaving a comment on a blog post entry isn’t what you’re looking for then here’s my business card!

And who are you reading? Well, this is me… or it was in 2016, anyway!

Breaking the MBR on every hard disk

Wed, 10 Feb 2016 19:46:00 -0500

I was reminded of a backblaze article about SMART numbers. This nudged me to look up the stats on my drives to see if any numbers had budged.

Let’s collect the data for processing:

  for a in /dev/sd?
  do
    smartctl -a $a > $a
  done

Spot the error.

I ran the code. Did an “ls”… and didn’t see any output. I started to panic a little… I didn’t just do what I think I just did… did I?

That’s right! I just overwrote the MBR on ALL my disks with smartctl output. 8*4Tb, 4*2Tb, 2*512Gb SSD and a 1Tb external disk. All broke.

Oh fuckety fuckety fuckety fuck.

At this point I was running around biting my knuckles. My heart was racing.

I switched to decaf. Used the walk to the coffee machine to force some thinking time.

The machine was still working. It hadn’t noticed.

I started to calm down…

The largest of the output was 10K long. Fortunately I don’t put the raw disks into RAID arrays; I create one large partition on each disk and use that. 10K is 20-ish sectors. The first partition on every disk was 2048 or 4096 sectors in, except for the external disk which was 64 sectors in. Yes, I waste 2Mbyte of disk this way…

But that wasted space may have saved me. I began to think that my data was safe. All I needed to do was recreate the MBR.

For that I needed the partition data. It must be in the kernel somewhere, right?

/proc/partitions was particularly useless. It tells me what partitions were there and the size… but not where it started.

Ah, /sys/block/$disk/$partition has two entries; “start” and “size”.

sda/sda1/start:2048
sda/sda1/size:16775168

That looks useful! I think I can recover… No partition type, but I know that :-)

Let’s grab all that data just in case the machine does crash and I lose it. This is my best chance of recovery.

A vague memory… GPT data is stored near the end of the disk… on a GPT disk the MBR is basically just used to “protect” the contents from non-GPT aware apps.

Bleh, parted bitches and moans. gdisk? Ooh, nice! It told me the MBR was barfed, and the GPT failed checksum… but gave me the option to use the GPT anyway. I took it… and it gave me the correct values! SAVE!

I was able to recover the 4*2Tb and 8*4Tb disks this way.

The other 3 disks were too small so I hadn’t bothered with GPT; they were just MBR partitioned.

Now let’s see what we can do with sda…

sda/sda1/start:2048
sda/sda1/size:16775168

sda/sda3/start:16777216
sda/sda3/size:983438000

fdisk, units, create primary 1, start at 2048, +16775168. create primary 3… oh, can’t start at 16777216. Partition 1 is 1 sector too large. Need to reduce the size by 1. Remember also to do that for Partion 3. Set the types to FD (Autoraid), make partition 1 active.

Repeat that for the other SSD. Similarly for the 1Tb disk.

Run grub-install against the two SSDs; they’re my boot mirrors.

Now the moment of truth…

reboot!

And… relax.

All the md RAID devices assembled correctly. The machine booted cleanly, as if nothing had gone wrong.

Of course there’s still junk in the MBR of most of the disks

% sudo dd if=/dev/sdb bs=1c count=256 | hdump -16
00000000  73 6D 61 72 74 63 74 6C 20 35 2E 34 33 20 32 30   smartctl 5.43 20
00000010  31 32 2D 30 36 2D 33 30 20 72 33 35 37 33 20 5B   12-06-30 r3573 [
00000020  78 38 36 5F 36 34 2D 6C 69 6E 75 78 2D 32 2E 36   x86_64-linux-2.6
00000030  2E 33 32 2D 35 37 33 2E 31 32 2E 31 2E 65 6C 36   .32-573.12.1.el6
00000040  2E 78 38 36 5F 36 34 5D 20 28 6C 6F 63 61 6C 20   .x86_64] (local
00000050  62 75 69 6C 64 29 0A 43 6F 70 79 72 69 67 68 74   build).Copyright
00000060  20 28 43 29 20 32 30 30 32 2D 31 32 20 62 79 20    (C) 2002-12 by
00000070  42 72 75 63 65 20 41 6C 6C 65 6E 2C 20 68 74 74   Bruce Allen, htt
00000080  70 3A 2F 2F 73 6D 61 72 74 6D 6F 6E 74 6F 6F 6C   p://smartmontool
00000090  73 2E 73 6F 75 72 63 65 66 6F 72 67 65 2E 6E 65   s.sourceforge.ne
000000a0  74 0A 0A 3D 3D 3D 20 53 54 41 52 54 20 4F 46 20   t..=== START OF
000000b0  49 4E 46 4F 52 4D 41 54 49 4F 4E 20 53 45 43 54   INFORMATION SECT
000000c0  49 4F 4E 20 3D 3D 3D 0A 4D 6F 64 65 6C 20 46 61   ION ===.Model Fa
000000d0  6D 69 6C 79 3A 20 20 20 20 20 53 65 61 67 61 74   mily:     Seagat
000000e0  65 20 42 61 72 72 61 63 75 64 61 20 47 72 65 65   e Barracuda Gree
000000f0  6E 20 28 41 64 76 2E 20 46 6F 72 6D 61 74 29 0A   n (Adv. Format).

But the important part is the partition table… and that’s just fine :-)

000001a0  62 79 74 65 73 20 5B 32 2E 30 30 20 54 42 5D 0A   bytes [2.00 TB].
000001b0  53 65 63 74 6F 72 20 53 00 00 00 00 00 00 00 00   Sector S........
000001c0  02 00 EE FF FF FF 01 00 00 00 AF 88 E0 E8 00 00   ................
000001d0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
000001e0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
000001f0  00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 AA   ..............U.

Lesson: stop doing shit as root. If I’d have done

sudo smartctl -a $a > $a

then the redirect would have failed and I would have saved myself a tonne of heartache.

Gullible

Mon, 24 Aug 2015 15:27:00 -0400

Coworker: i’ve found that networking/factime can pay dividends

CW: *face time

Me: “factime” - that was the early name for “bullet time”, but they realised “fast and circular” didn’t sound as cool.

CW: stephen you ever play trivial pursuit and if so how’d you do?

CW: or maybe jeopardy

Me: Umm, you really shouldn’t believe everything I write :-)

CW: lol

CW2: Stephen, have you ever played balderdash? And if so, how’d you do? ;-)

DHS redux

Sat, 04 Jul 2015 07:30:00 -0400

So it looks like those scans were coming from NCATS.

This is only meant to scan networks associated with the Federal government. I’m guessing there was a misconfiguration, somewhere, ‘cos Panix tell me they never requested any scans of their network :-)

Through a friend I contacted their SOC. I saw another scan yesterday and escalated. They just replied and told me that they’ve removed the IP ranges from their config.

On the plus side, the only thing flagged by their scanning was that I had TRACE enabled on my web site. Everything else looked good :-)

Huh, the department of homeland security is attacking me?

Sat, 04 Jul 2015 07:30:00 -0400

Either the DHS is attacking me, or else they’ve got compromised computers…

In my logs I see 1147 attempts from 64.69.57.20 to my web server; e.g.

64.69.57.20 - - [03/Jul/2015:00:40:32 -0400] "\x16\x03\x01" 501 295 "-" "-"
64.69.57.20 - - [03/Jul/2015:00:40:40 -0400] "GNUTELLA CONNECT/0.6" 400 306 "-" "-"
64.69.57.20 - - [03/Jul/2015:00:40:41 -0400] "GET http://rfi.nessus.org/check_proxy.html HTTP/1.0" 404 293 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"
64.69.57.20 - - [03/Jul/2015:00:40:42 -0400] "ABKJFC / HTTP/1.1" 501 303 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"
64.69.57.20 - - [03/Jul/2015:00:40:42 -0400] "GET /aboutprinter.html HTTP/1.1" 404 300 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"
64.69.57.20 - - [03/Jul/2015:00:40:42 -0400] "GET /properties/configuration.php?tab=Status HTTP/1.1" 404 311 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"
64.69.57.20 - - [03/Jul/2015:00:40:52 -0400] "GET /etc/passwd HTTP/1.1" 404 293 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"
64.69.57.20 - - [03/Jul/2015:00:41:09 -0400] "GET /?<meta%20http-equiv=Set-Cookie%20content=%22testrdhw=3733%22> HTTP/1.1" 200 6059 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)"

logwatch summary:

 Requests with error response codes
    400 Bad Request
       %.: 2 Time(s)
       %5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e%2e%5c%2e ... winnt%5cwin.ini: 2 Time(s)
       .: 2 Time(s)
       ../../../../../../../../../../../../windows/win.ini: 2 Time(s)
       ../../../../../../../../../../../../winnt/win.ini: 2 Time(s)
       ..\\..\\..\\..\\..\\..\\..\\..\\..\\..\\windows\\win.ini: 2 Time(s)
       ..\\..\\..\\..\\..\\..\\..\\..\\..\\..\\winnt\\win.ini: 2 Time(s)
       ..\\..\\..\\..\\..\\..\\windows\\win.ini: 2 Time(s)
       ..\\..\\..\\..\\..\\..\\winnt\\win.ini: 2 Time(s)
       ./././././././././././././././././././././ ... ../../../../../: 2 Time(s)
       .\\.\\.\\.\\.\\.\\.\\.\\.\\.\\/windows/win.ini: 2 Time(s)
       .\\.\\.\\.\\.\\.\\.\\.\\.\\.\\/winnt/win.ini: 2 Time(s)
       /: 9 Time(s)
       /%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e ... e%2e/etc/passwd: 2 Time(s)
       /%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e ... e/winnt/win.ini: 2 Time(s)
       /%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e ... windows/win.ini: 2 Time(s)
       /%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd: 2 Time(s)
       /%NETHOOD%/: 2 Time(s)
       /%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%u ... e/winnt/win.ini: 2 Time(s)
       /%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%u ... ff0e/etc/passwd: 2 Time(s)
       /%uff0e%uff0e/%uff0e%uff0e/%uff0e%uff0e/%u ... windows/win.ini: 2 Time(s)
       /../../../../../../../../../../../../etc/passwd: 2 Time(s)
       /../../../../../../../../../../../../windows/win.ini: 2 Time(s)
       /../../../../../../../../../../../../winnt/win.ini: 2 Time(s)
       /./../../../../../../../../../../../etc/passwd: 2 Time(s)
       /././..: 2 Time(s)
       /././././././../../../../../etc/passwd: 2 Time(s)
       /././././././../../../../../windows/win.ini: 2 Time(s)
       /././././././../../../../../winnt/win.ini: 2 Time(s)
       //../../../../../../../../../../../../etc/passwd: 2 Time(s)
       /password: 2 Time(s)
       /tmUnblock.cgi: 1 Time(s)
       1435898497:@166.84.7.9/: 4 Time(s)
       : 1 Time(s)
       : 1 Time(s)
       : 1 Time(s)
       : 1 Time(s)
       CONNECT/0.4: 2 Time(s)
       CONNECT/0.6: 2 Time(s)
       c:\\boot.ini: 2 Time(s)
       invalid: 2 Time(s)
    403 Forbidden
       /: 1 Time(s)
       /.htaccess.1: 2 Time(s)
       /.htaccess.bak: 2 Time(s)
       /.htaccess.copy: 2 Time(s)
       /.htaccess.old: 2 Time(s)
       /.htaccess.tmp: 2 Time(s)
       /.htaccess.~1~: 2 Time(s)
       /.htaccess~: 2 Time(s)
    404 Not Found SUMMARY - 733 URLs, total: 1483 Time(s)
    405 Method Not Allowed
       /: 4 Time(s)
       /gprvpp1.html: 1 Time(s)
       /pevwoo1.html: 1 Time(s)
    417 Expectation Failed
       /: 1 Time(s)
    501 Not Implemented
       *: 2 Time(s)
       /: 4 Time(s)
       null: 5 Time(s)

The Nessus proxy check line makes me think this might be a generic scan… but why my machine?

They didn’t stop there… I have SSHD running on a non-standard port. If someone attempts to connect too frequently then they get blocked (simple iptables rule). I can see 6 dropped packets from the same SRC=64.69.57.20 to my SSH port.

Didn’t stop there, either. DNS attempts?

client 64.69.57.20 bad zone transfer request: 'dastardly.spuddy.org/IN': non-authoritative zone (NOTAUTH): 1 Time(s)
client 64.69.57.20 bad zone transfer request: 'org/IN': non-authoritative zone (NOTAUTH): 1 Time(s)
client 64.69.57.20 bad zone transfer request: 'ssl.spuddy.org/IN': non-authoritative zone (NOTAUTH): 1 Time(s)
client 64.69.57.20 update forwarding 'spuddy.org/IN' denied: 1 Time(s)
client 64.69.57.20 query (cache) 'example.com/A/IN' denied
client 64.69.57.20 query (cache) '\.\./nessus/A/IN' denied

Looks like also some port scans, ‘cos I can see “rsync” (started from xinetd) being woken up (but it rejects them access).

And, from another machine on the same network, SMTP attacks!

CONNECT from unknown[64.69.57.28]: 503 5.5.0 : Client host rejected: Improper use of SMTP command pipelining; proto=SMTP
non-SMTP command from unknown[64.69.57.28]: GET / HTTP/1.0 : 1 Time(s)
non-SMTP command from unknown[64.69.57.28]: GET /nice%20ports%2C/Tri%6Eity.txt%2ebak HTTP/1.0 : 1 Time(s)                
non-SMTP command from unknown[64.69.57.28]: Via: SIP/2.0/TCP nm;branch=foo: 1 Time(s)

(66 attempts against SMTP)

OK, OK, this all looks like an “out of the box” type scan from some misconfigured security tool. But it’s funny that it’s the DHS!

A joke I learned in 1978

Tue, 26 May 2015 14:48:00 -0400

When I was 10 my teacher told us this joke.

So in the old Wild West… General Custer was chasing Sitting Bull. He had his cavalry and supply wagons and soldiers. They’d been chasing Sitting Bull for days and the weather was starting to go bad. A lieutentant came to Custer and told him “Sir, the men are tired, the weather is bad, we need to rest.” Custer thought about it and replied “OK, I’ve been here before; a couple of miles ahead is a valley where we can make camp”.

So they set up camp. But this valley focused the wind and the cavalry were miserable. The lieutenant complained again to Custer; “As I said, I’ve been here before. At the far end of the valley there is a grove of hazelnut trees; you can cut those down and make huts to protect us from the wind”.

Which they did. But hazelnut branches are twisty and there were lots of gaps in the walls for the wind to fit though. Still the cavalry shivered. A third time the lieutenant went to his General; “The rear wagon is full of chocolate; melt that and poor it over the huts. That will seal the holes”.

Finally the cavalry were able to rest.

Now high above them, on the valley’s edge, Standing Bull (son of Sitting Bull) watched on. He saw the cavalry do all this, and was confused. He reported to his father.

“So, my son! Tell me what my old enemy General Custer is doing now.”

And so Standing Bull told him of the cavalry, of the wagon train, of the cutting down of trees, the building…

“Father, I don’t understand these actions. Has your enemy gone mad?”

“No my son, it is very simple. (singing) Huts, oh hazel huts, cavalry make them and they cover them with chocolate!”

Youtube

Some good Voyager

Thu, 30 Apr 2015 22:15:02 -0400

So I wrote that Voyager was the worst ST ever.

I still agree with that, because of the overall structure.

This doesn’t deny that some episodes were good though.

Viewing Voyager as a “old school story telling” structure allows you to see each episode as an individual story.

One really good one is “Child’s Play” (season 6). This is a “7 of 9 episode” (one of the downsides of Voyager was “harry episode”, “tom episode”, “tuvok episode” etc etc), but it was one of the better standalone episodes, talking to what it means to be an individual, to have self-agency. It’s a well done story… but it only works because of the episodic nature; next week it doesn’t matter.

ETA

And then the beginning of the next episode has Janeway telling Kim $foo, who tells Seven $bar, who creates a PHYSICAL order for Belana, who hands that to RANDOM person who walks a mile to UNKNOWN person who clicks a button.

WTF?

That’s just stupid process.

And I work in a company with stupid processes. THIS chain of events is… wrong.

Why Star Trek Voyager was the worst of all

Wed, 22 Apr 2015 20:53:00 -0400

“Voyager” has a reputation for being awful. And I agree with this. But maybe not for the reason commonly cited.

It’s an article of faith amongst some people that Voyager is panned simply because it had a woman in the captain’s chair. I have no doubt that there are plenty of Neanderthals out there who do have this stupid knee jerk reaction. I remember some of the media speculation around having a female captain, and how the audience would take it. I didn’t understand why this was a big deal. Hmm, this was 1995 so I would have been 25 or 27. I’d grown up with Heinlein and others. I had a strong mother (smarter than my father). I knew women were just as capable as men. So why wouldn’t there be a female captain? (Pretty sure we’d seem them in ST:TNG as well, just not as the lead character).

The show started and… ugh, the voice. Janeway’s voice was grating. But… I got used to it. Just like having to get used to a new Doctor Who, I got used to Janeway’s voice.

Now ST:DS9 had already been running for a couple of years so the idea of a combined crew not uniquely Star Fleet was well established. The combined Maquis/Fleet would have worked. But that’s not what we got; we got a “determined Star Fleet” ship, with some of the tension being how the Maquis fit in. Disappointment #1.

But that’s not the real problem I had with the show.

Babylon 5 had just completed its run. It changed how SciFi TV story telling was done. Good shows were no longer episodic in nature; they were one linear story told over multiple episodes. Things were revealed, consequences had actions. We didn’t ‘reset at the end of the episode’.

Now previous Star Trek stories were definitely episodic. TOS and TNG were products of their time; everything was episodic. DS9 started that way, but midway through they realised “hey, story arcs are good!” and built one around the war with the Dominion. The characters grew and changed; actions had consequences that continued on. The story now had “state”.

Voyager started episodic… and continued episodic. It never grew; it never changed. “Whee! we got a super-duper transwarp drive! Home in 2 weeks! Oh, but we can’t use it ever again ‘cos it’d break the ship. Back to 69 more years of travel…“. Seriously, that happened!

And, somehow, despite being beyond the reach of Star Fleet and short on energy (hence needing Neelix to create fresh food) there’s always more photon torpedoes, spare parts. That’s all replicated? Ha, no.

There’s one episode in season 5 where Tom Paris gets demoted from Lt down to Ensign. Now, other than his title changing in every episode afterwards everything else is exactly the same. There’s no change in the character, in the way Janeway acts towards him. There’s no real consequence. It’s just a name change.

This is why Voyager was the worst Star Trek of all; it’s method of story telling was 5 years or more out of date before it started. We’d seen the future (Babylon 5), and Voyager was a disappointment in comparison. If it had been done 7 years earlier then it would have stood up pretty well (definitely better than Enterprise in many respects; that bloody Temporal War…), but it wasn’t. It was old school story telling, made even more prominent by DS9 overlapping and using the new school.

I finally bought the Voyager DVDs (they were on sale, cheap) and I’m in the middle of watching them (just about to finish season 5). As an episodic series I like it. The distance of time can make me forget that I expected more, I expected better. If I mentally place it “pre-B5” then I can forgive the episodic sins (except the infinite photon supply!). It’s good harmless fun. But I know that I can watch 50 more episodes and nothing’s gonna change (until the series finale when “magic happens”). You have to take each episode on their own and not expect anything more than 45 minutes of entertainment.

How does the web still work?

Tue, 24 Mar 2015 08:45:51 -0400

I hit a web page which, naturally, refused to work properly. So I looked at the NoScript report. This one page ws pulling in scripts from (hand-typed so maybe tpyos)

adobedtm.com
cdna-assets.com
chartbeat.com
cloudfront.net
criteo.com
disqus.com
disquscdn.com
doubleclick.net
dunhilltraveldeals.com
effectivemeasure.net
facebook.com
gigya.com
google.com
googlesyndication.com
googletagservices.com
imrworldwide.com
inksinmedia.com
krxd.net
mediavoice.com
mmcdn.us
ooyala.com
optimizely.com
outbrain.com
parsly.com
quantserve.com
qubitproducts.com
revsci.net
scorecardresearch.com
skimresources.com
visualrevenue.com
whistleout.com

Boggle!

So You Start Server

Tue, 11 Mar 2014 22:34:30 -0500

I have a linode and a Panix v-colo. These servers do everything I want.

But OVH do a physical server rental program. It’s not necessarily the best server in the world, but it’s pretty good. The So You Start(SYS) server starts at around $50 month. Now a linode is $20/month. 10% discount for paying 1 yr in advance, so $18/month.

To compare; for $18/month I get a 8vCPU, 1Gb RAM, 48Gb of disk on my linode. On SoYouStart for $50 I get a quad-core Xeon E3-1245 (HT==> 8 CPU), with 32Gb RAM and 2*2Tb disk.

You look at the numbers and a SYS server knocks linode out of the park.

Except I really don’t need that much processing power. Which is why alcohol got involved and I bought one.

I called the new machine duckula because it’s sucking money out of me for no good reason, just like a vampire.

But what do you get?

It’s actually pretty impressive. SYS is a cut-down service from the standard OVH service so the control panel is very limited. “Redeploy an OS”, “reboot” and “rDNS” are pretty much all you can do. The “deploy OS” has a small amount of flexibility, but not much. Compared to Linode or even Panix the SYS control panel is crap. In particular you don’t get console access.

You also need to be aware; the SYS server has some back doors installed. Well… If you google “ovh backdoor” you’ll see a lot of posts. I can’t verify all of them; I’d overridden the root authorised_keys stuff and sshd_config pretty early on in my setup so I don’t know if they push keys in the standard build. They do have a per-minute cron that pushes a fuck-tonne of crap about your server into their monitoring system.

Essentially, their standard build is not vendor standard. There are additional components and potential backdoors ‘for support purposes’. I think I’ve turned them all off.

But that seems to be where the negatives stop… mostly

What you get for $50 a month… 32Gb RAM, 2*2Tb disk in RAID1, ECC RAM, quad core Xeon (HT), IPv4, IPv6/64, rDNS.

That’s a pretty impressive spec machine. Except for disk space, that’s better than my home machine (more memory, ECC memory, better CPU). If you wanted to buy such a machine and colo it then it’d probably cost you more.

My gut feeling about SYS is “ugh, they backdoor…”, but other than that I’m feeling this isn’t a bad offering and seems pretty competitive in terms of what you get for your buck.

Do I need such a machine? No. But… I’ve already had one benefit; I can use Linux kvm to host a small server for a friend!

Historical Java

Fri, 28 Feb 2014 12:23:32 -0500

Discussions elsewhere (in more than one place; coincidence?) reminded me that in the long long distant past I was “hot” java programmer (“hotjava”, geddit? Ah shaddup!). I actually wrote what I believe was the first website Java applet sold commercially in the UK, back in 1995. It was a teaser site for the SciFi Channel (which was due to launch around then) called “zorg”. We got involved ‘cos we were Sun partners and the client had gone to Sun asking for some Java development.

The ad company (Star Interactive) had the concept of “aliens were trying to communicate with us and learn our language”. So they paid for teaser icons to be placed on some high traffic UK web sites (with no explanation; just some “alien looking writing”) that linked to the zorg site. There you were presented with a sort of rip-off of Stargate as a Java applet. There was a pretty neat graphic of a square alien device with writing on it. And a “throbbing” type sound. If you clicked on the writing then the writing would flash at random and various of the writings would stay lit. The sound increased in speed and pitch. Click again, the same thing happens. Click a third time and…

If you got it wrong the screen flashed red and you were transported to some foreign web site, such as the Swedish Phone Directory. Get it almost right and the sound effects would ramp up and then stop, and you’d be transported to one of the teaser sites. If you got it correct and the sounds effects would ramp up, a vortex would open and then you got passed onto the second game (which was a traditional HTML game; no more java). If you won all three games then you got to an “egg” image with partially decoded alien writing. As the channel launch date came closer more of the writing was decoded.

The rule was pretty simple; each icon was numbered. If you clicked on todays number then you failed (so the solution was different each day). If you clicked either side of today then you won. Else you almost won. It worked :-)

This was very early in Java history. Sun HotJava Alpha 3 was the only java browser. Netscape 2.0b1 arrived and this had a different java model. And every version of the Netscape betas changed something in java each time. So the index page turned into a quick CGI that went to whatever code version was required for your browser (or a html based one for non-java people). But it worked and we got a few hits.

It was fun. I hadn’t even seen the Stargate film at the time, but I think I did a pretty good immitation :-) Gamelan rated the site as “cool” with a comment such as “we don’t know what it does, but it’s fun!” Later, after the launch was complete I added a comment explaining what the site was about. I doubt anyone would have got the ad peoples “concept”!

There are few mentions of it anymore. Unfortunately I’ve lost the source :-(

This led me into delving through old directories of crud I’ve accumulated over the years. And I found my very first piece of Java, written for HotJava Alpha3. This was a simple drag’n’drop system; click on an icon and drag it around and release it. I also added the icon changing when you clicked on it and some simple sound effects. A slightly newer version still works on current browsers, and can be found here – or, at least did work until browsers such as chrome dropped java support totally.

One big difference between HotJava Alpha3 and Netscape can be seen in the current version. First time you click on something there’s a delay. This is because Netscape delays actual loading of data (such as images and sounds) until it’s needed, whereas HotJava would download immediately. So in HotJava,

Image im1 = getImage(s + "/T" + i + ".gif");

would result in an immediate hit on the webserver; in Netscape it didn’t. So when I wrote the zorg game I had to pre-cache all the data to improve “game play”.

And then I didn’t touch Java at all for 7 or 8 more years, when I wrote a simple graphing program (which would allow arbitrary zooming and stuff) for work. And I haven’t touched it since.

My web design skills are basically stuck back in 1996 with HTML2 and tables. One day I should really learn CSS and Javascript and DHTML and all the neato stuff that’s around. Until then…

Scripts

Mon, 09 Dec 2013 14:47:02 -0500

I can code, but mostly I write quick scripts for various purposes. I’ve become pretty infamous at work for the horrendous one-off one-line pipelines I create! I try and make actual scripts a little more readable…

denon_demon
My Denon receiver has a serial port. This serial port can be used to control and report on what the receiver is doing. So if I send it "PW?" then it will report on the power state. This daemon (written for MacOS) will listen for commands on a pipe and communicate with the receiver and write out status entries so it is easy for a program to see what is going on. It also writes out a log file.
```
Wed Feb 22 21:21:27 2012 status.Volume = 47 -33.0dB
Wed Feb 22 21:21:30 2012 status.Volume = 465 -33.5dB
Wed Feb 22 21:21:59 2012 status.MainSurround = STEREO
Wed Feb 22 21:22:00 2012 status.MainSurround = STEREO
Wed Feb 22 21:23:58 2012 status.MainSurround = STEREO
Wed Feb 22 21:24:02 2012 status.Power = STANDBY
Wed Feb 22 21:24:02 2012 status.MainZone = OFF
```

denon
A simple command client to talk to the pipe above, and report on the status of the receiver

% denon volume -25
          Input: TIVO
   MainSurround: DOLBY PL2 C
       MainZone: ON
           Mute: OFF
          Power: ON
         Volume: 55 -25.0dB
         Z2Mute: OFF
       Z2Volume: 99 ---.-dB
          Zone2: OFF
     Zone2Input: SOURCE

phone
Pulling data from Google contacts using LWP::Simple and XML::XPath rather than WWW::Google::Contacts, which has a horrendous dependency path. Since we're only reading data here we can do things more easily.
phone_droid3
I had a Droid 3 phone. This script will scp the contacts.db and then use DBD::Sqlite to read the data and present it in text format. Why am I doing this? 'Cos I'm stupid :-)
movemouse.c
OK, this is a C program and not a script, but it's small enough so... Basically moves the OS-X mouse cursor to a given co-ordinate. This is because my Mac Mini will reset the cursor to top-left frequently and when I play DVDs then this stops the application menu bar going away, so I use this to move the mouse first.
dvdplay
This is a a bash script that creates applescript to start DVD Player playing a VIDEO_TS directory.
dvd
A script that generates various small applescripts to report/control on DVD Player. Uses dvdplay to start a new DVD. So I can do "dvd pause" to pause playback.
play (iTunes 10 version)
Shell script to create applescript to create an iTunes play list to play the mp3s listed on the command line
itunes (iTunes 10 version)
A script that generates various small applescripts to report/control on iTunes. So I can do "itunes dj" to play the "iTunes DJ" playlist. Unfortunately the latest version of iTunes has broken a lot of functionality; the applescript calls don't work properly any more and a number of controls no longer have directly accessible function calls.
itunes
play
A version of the "play" and "itunes" scripts that work with Itunes 12. This includes some kludge workarounds for manipulating the shuffle and repeat settings.
airplay
A craptastic script that tries to turn on/off airplay on iTunes so that I can listen to the music on a secondary receiver. Apple don't seem to expose the AirPlay interface so this pretends to click buttons and stuff. Ugh.
virt_build
A BUILD script for KVM `virsh` to create VMs. It's a wrapper around `virt-install` to create a disk image or LVM, define the machine, kickstart boot it. Additional disks can be added to images using virt_add_disk. The CentOS 6 and CentOS 7 kickstart files.

Google Authenticator

Mon, 23 Sep 2013 07:41:48 -0400

So I decided to play a little bit with google authenticator on my systems that are visible to the internet. ie my linode, Panix v-colo and ‘bastion’ host at home.

The way sshd works, if you authenticate with public keys then PAM “auth” doesn’t seem to get called. So this is pretty much for “ChallengeResponse” (instead of “password”) authentication. Which makes it great for my need; if I’m coming from one of my own machines with my SSH key then I’m not impacted. If I’m coming in from someone elses machine then I need to know my password and the one-time-password.

Now on these hosts the only account’s with a valid password are my own and root and sshd is configured to not allow passwords for root (ssh keys only). I also run sshd on a non-standard port and this has stopped dead random password hack attempts. And no one is trying to attack me (I don’t run any popular blog; I’m not a bank, I’m not a target). So, in practice, google authenticator doesn’t really add anything for me.

But it was interesting to play with.

(Of course if I’m away from my machine and my phone then I have a problem!)

Teaching myself javascript

Sun, 06 Jan 2013 11:50:26 -0500

My web design skills are a little on the old side. Probably frozen around 1997; Netscape Navigator 2 would probably render most of my stuff properly.

As part of the “make work” project I did last month I wanted to try something new. With the existing version if you clicked on a link to get cover-art then it would just be a simple link. You see the image then press “BACK”. This worked well for years, but some mobile devices don’t cache too well. And even if they do then the rendering time can be quite large. One obvious option would be to open in a new window (tab, whatever), but I thought I’d try to do “popups”. It turns out the term of art is “lightbox”.

Armed with that I found Lightbox 2. It took 5 minutes to integrate that into my page. And it worked. Trouble is that it required jQuery so the whole thing took over 100K. Considering the original page was less than 80K, I’ve over doubled my download size.

Also I didn’t learn anything from this. So a solution that worked, but was unsatisfying.

I did some more searching around. I found some “10 line” solutions, which looked fun… but they all worked on the concept of setting the image to be invisible and then show it on demand. Which, given I was planning on 1200 images, would have meant pre-loading all 1200 images on the page in the off-chance of clicking on one or two links.

But I did learn from these things.

My goal was to have 1 image; change what the image displayed; move it to the right place; then display it.

It turns out to be a simple concept, but slightly messy.

The core code is a small CSS:

  <style>
    .white_content {
      display: none;
      position: absolute;
      padding: 16px;
      background-color:#eae9d4;
      z-index:1000;
      overflow: auto;
    }
  </style>

And then some javascript

<script type="text/javascript">
var oldfocus = null;

function set_img(i,e)
{
  document.images['popup'].src=i;
  oldfocus=e;
  var x=e.offsetLeft+e.offsetParent.offsetLeft-20;
  var y=e.offsetTop+e.offsetParent.offsetTop-200;
  if (x<0) { x=0; }
  if (y<0) { y=0; }
  document.getElementById('light').style.left=x + 'px';
  document.getElementById('light').style.top=y + 'px';
  document.getElementById('light').style.display='block';
  document.getElementById('poplink').focus();
  return false;
}
function hide()
{
  document.getElementById('light').style.display='none';
  oldfocus.focus();
}
</script>

Add in a DIV to hold the image:

<div id="light" class="white_content">
<a id='poplink' href='javascript:void(0)' onclick="hide()"><img border=0 name=popup src="images/spacer.gif"></a>
</div>

And now it’s easy to call:

<a target=_blank href="images/sp-sweh.gif" onClick="return set_img('images/sp-sweh.gif',this)">South Park</a>

Originally I wrote the code to track X,Y position by catching mousedown events. But keyboard navigation (eg TABbing to a link and pressing RETURN) doesn’t cause mousedown events, so no X,Y is captured. So this code tries to work out where the link is on the page and use that. It turns out the “offsetParent” stuff should really be chained; eg a link inside a table has the link offset relative to the cell; the cell offset relative to the table; the table offset relative to the page. So that code got rewritten as a loop in my final version. But, basically, that’s a lightbox with 33 lines of code.

Next I wanted to see if I could render HTML rather than show images. Turns out it’s mostly the same thing; Using the “innterHTML” attribute allows a DIV to render almost anything, not just images.

The final solution…Image popups and HTML popups

Exelink

Fri, 01 Jun 2012 13:44:03 -0400

In the early 90s I was learning DOS and Unix. One thing I really liked about Unix was the concept of “symbolic links”. It meant you could install programs in their own directories and symlink the executable to somewhere in your $PATH. DOS couldn’t do that. So you ended up with horrendously long path names. And DOS had a 120 character limit. Ugh.

So I wrote a TSR, called “exelink” which kludged it. I used it for years until I gave up on DOS and became a Unix person. Even sold a copy to “PC Plus” for a cover disk.

Annoyingly I lost the code. Random searches of the internet didn’t find anything useful. Except a random search, today, found a usenet posting where I mentioned that I’d uploaded it to SIMTEL20. Huh. And that was enough to find it!

And, heh, here’s the README file…

                                  EXELINK.COM
                                  ===========
 
IMPORTANT NOTE:  Needs DOS 3.0 or greater
 
#include <std_disclaimer.h>
 
All the code in this package is my own work.  No responsibility will be
taken for loss of data or damage caused by running this program.  This
program has been tested and I use it myself, so if you do come across
any problems then let me know!
 
The program can be freely distributed.  There are no restrictions on use.
Just keep my name on it please!
The program is "postcardware" - send me a postcard if you like :-)
 
Introduction
============
A big problem with Hard discs is that they can store so much data!
This isn't as silly as it sounds; if you have four or five applications
on your disc (maybe Pascal, C, Assembler, Word Processor, Norton etc)
and wish to keep some sort of organisation do your disc then your PATH
variable gets a little large.  Until you reach the dreaded 127
character limit.  Although programs exist that enable you to get up to
255 characters in the PATH, this is a bit long and unwieldy, and can
make you computer spend too much time skipping around directories just
to find the program you want.  I had this problem and decided to find a
better solution, and EXELINK is it (well I think its better!)
 
What EXELINK does
=================
NOTE: EXELINK will not work with .BAT files since the command shell
interprets them internally and doesn't call the DOS 'exec' routines.
Only routines that call 'exec' (eg .EXE and .COM files, Overlays?)
work.
 
When you execute a command, MS-DOS needs to find where the file is that
you are trying to run.  It first looks in the current directory, then
in the first directory on the PATH, then the next, then the next...
until it finds what you wanted.
 
eg if you have your path set up as
c:\dos;c:\app\turbop;c:\app\turboc;c:\app\tasm;c:\app\wp;c:\app\norton
then typing the command 'nu'  (for norton utilities) will involve
looking for
     .\NU.???
     c:\dos\NU.???
     c:\app\turbop\NU.???
     c:\app\turboc\NU.???
     c:\app\tasm\NU.???
     c:\app\wp\NU.???
     c:\app\norton\NU.???
which it will eventually find.  Any files in any of these directories
which are not executable will slow down the searching...
Of course an easy way to speed up access to NU would be to put it at
the front of the PATH, at the expense of some other program!
 
EXELINK sets aside directory which you include on your PATH (for speed
you put it at the beginning) and this contains a set of files which
appear to DOS to be executable.  Instead of being run though, EXELINK
reads the contents of this file and gets the filename of the REAL
program from there.  Since the link directory is at the beginning of
the PATH and there are only executables it will find the link file very
quickly.
In the example above, you could now have the following setup:
PATH=c:\links;c:\dos
and searching would be
     .\NU.???
     c:\links\NU.???  <- found!  Open the file, read its contents and
                         execute!
 
Less searching is done, executables are found quicker, PATH is shorter
and less disk head movement is used.
 
But what is this I hear?  You can do this by creating a BATch files which
calls the real program?  Almost true, but there are a  few problems with
batch files:
 (1) There is difficulty passing an unknown number of parameters.  A batch
     file such as
     @c:\app\wp %1 %2 %3 %4 %5
     can be taken as meaning that is passes exactly five parameters, some
     of which are NULL (GEM programs can suffer from this problem)
 (2) It only works from the DOS prompt - a program such as MAKE is
     unable to use BATch files unless you specifically call COMMAND.COM
     which is slower and takes more memory.
 (3) It is an interpreted file.  DOS opens the BATch file, reads
     a line.  If it executes an external program, closes the BATch
     file, executes the command, opens the BATch file, etc.  Lots of
     disc accessing...
 
The only real alternative would be to write a specialised Pascal or C
program which just executed the real version with all the parameters
passed on.  This will work in all cases, but has the problem that you
need to compile a new link program for each new executable you want
linked, and even the simplest compiled program takes about 2.5k -
wasting 4k of disc space!
example in Turbo C 2.01:
 
main(int argc,char *argv[], char **envp)
{
  argv[0]="real_name.exe";
  execve("real_path_name.exe",argv,envp);
}
 
Which would then need compiling, and even in the TINY memory model
takes over 3k!
 
Using EXELINK you need to create a 1 line ASCII file which you can do
with the COPY command.
How EXELINK works
=================
EXELINK is a Terminate and Stay Resident (TSR) program.  While resident
it takes an astounding 400 bytes of memory!  The only vector it
attaches to is INT 21h and any call that is not Function 4Bh (exec) is
passed directly onto the original handler.  To cut down on resident size
it frees the environment block passes to it by DOS, and relocates itself
over the PSP, which is no longer needed once the program has gone resident.
 
If the call is an 'exec' then the pathname of the program to be called
is worked out.  If this program is in the EXELINK directory (by default
C:\LINKS) then the file is opened for reading and the contents of the
file (up to 127 characters) are read in.  A little processing is done
to convert this to an ASCIIZ string and then the original routine
continues with this new name.  All registers are preserved through this
except for DS:DX which point to the filenames so any options passed to
'exec' are preserved.  Also, as far as the called application is concerned,
it was run by typing the full pathname of the program, so that programs that
use the command line (eg WordPerfect) to find the directory it is running
in work properly.
 
 
Requirements
============
EXELINK makes use of an internal DOS function:  INT 21h Function 60h -
CANONICALIZE FILENAME OR PATH
This is only available on MS-DOS 3.0 onwards, and I am not sure if this is
a fully documented call.
 
EXELINK has been tested with:
     PC-DOS 3.20
     vanilla MS-DOS 3.30
     Atari PC2 specific MS-DOS 3.21
     Tri-Gem MS-DOS 3.30
     OEM MS-DOS 5.0
     4DOS v3.01, v3.02, v3.02a, v4.0, v4.01 (a good COMMAND.COM replacement)
     Windows 3.0, 3.1
     DesqView 386 v2.3, v2.4
     Various applications.
 
Known incompatibilities:
     MKS Tools 3.1c version of the Korn-Shell.
       I have had a lot of problems with the Korn Shell, from TSR's to
       bog-standard applications.  I am unsure as to why this shell fails
       to work with EXELINK since it seems to support INT21/AX=60h
       I have not tested with later versions of MKS tools.
 
     Turbo C exec() call.
       Turbo C seems to try to open the executable and read the header
       contents to determine the file type.  Of course, these files are
       ASCII and so TurboC returns "Invalid exec type" :-(
       This also affects Borland C++ 3.1 (the Run Time Library Sources
       show why Borland do this - to enable exec() to overlay better)
 
     Windows 3.0, 3.1.
       I think Windows also tries to open the file to determine
       if a file is a DOS or a Windows executable.  I am not sure.
       However,  EXELINK works happily inside a DOS window.
     DesqView 386.
       It appears that DesqView has its own exec() replacement, and so
       each window that you wish to use EXELINK in must have its own
       copy of EXELINK loaded.  Once this has happened, it works happily
       inside that window.
     Word Perfect for Windows 5.1
       Word Perfect seems to search the PATH to find where your windows
       system resides, so it knows where to put the INI GRP DLL files.
       If WIN.COM is in the links directory then WP4W assumes this is where
       windows lives!  Silly Word Perfect.....before installing this program
       you should modify your PATH so that it has the real windows directory
       at the front.  This is only needed to work around the install problem,
       and is not needed at runtime.
 
    Most problems appear to boil down to the program trying to open the
    executable as a file to determine its type.  In Windows this is only
    a problem at the menu level (normally!) and creating an icon
    pointing to the real program is no major problem.  With DesqView
    creating an "Open Program" item pointing to the real program solves the
    problem as well, with a special "DOS" shell running exelink when that is
    needed.
 
    I had debated with the idea of extending exelink so that the open() call
    is also linked, but decided against this for various reasons, mainly
    that I would also need to trap filesize() and other related calls to
    maintain consistency, which would make exelink too large and unwieldly.
    Finally there would also need to be a way to deactivate exelink so that
    the link file could be editted!
 
What if the link points to a non-existent file?
   With 4DOS if you have an link pointing to a non-existent file then 4DOS
   will give the error: Invalid path "link_name".  The name it gives is
   the file in the LINK directory, so you can TYPE this file and find out
   why it doesn't work.  Under command.com 3.3 the error you get reads
   "EXE failure".  Under command.com 5.0 the error is:
   Cannot Execute "link_name".
 
Installation
============
1) Put EXELINK.COM in a directory on your PATH
2) Add the line
   [drive:][directory]EXELINK
   to your autoexec.bat file.  It is recommended that this is one of the
   first TSR's loaded so that the links are active as soon as possible.
   Then either reboot to run the new autoexec sequence, or run EXELINK
   by hand to be able to use it now.  EXELINK will load high with no problem
   (at least it does with QEMM).
3) Put the link directory at the beginning of your path
4) create the link directory.
5) create all the links needed.  Remember, a link file is just an ASCII
   file which contains the filename of the program you really want run.
Helpful Hints
=============
Applications called via EXELINK can live on any disk accessible by DOS.
There may be problems with 'join' and 'assign'.  The function INT21/AX=60h
used to test whether the called program is in the link directory returns the
filename as if there were no join's or assign's.  Your milage may vary.
 
Using 4DOS a nice utility that goes well is a small alias:
     alias addlink `for %add in (%&) echo %@full[%add]
            > c:\links\%@name[%add].%@ext[%add]`
[all on the same line of course!]
Alternatively you could create ADDLINK.BTM which performs the same
thing.  Then you could do something like
     cdd c:\app\turbop
     addlink *.com *.exe
which would create links for all .exe and .com files in the turbop
directory, enabling you to remove c:\app\turbop from the PATH.
If you have 'noclobber' set (setdos /n1) then any link that already exists
will not be overwritten.
 
Under MS-DOS command.com you could do the following:
create a file addlink.bat containing
     echo %1\%2 > c:\links\%2
Then you could do
     cd \app\turbop
     for %a in (*.com *.exe) do addlink c:\app\turbop %a
 
WARNING:  This method would erase any old link of the same name, as would
the 4DOS version without 'noclobber' set.
 
The link files are PURE ASCII and can be edited using any editor
from Edlin upwards.  The first character with a code of 32 or less counts
as the end of the string, so a ^Z or CR or LF are all ways of ending the
link file.   Up to 127 characters are allowed in the filename
on one line.  Anything longer than that will be truncated.  Filenames
must be FULLY QUALIFIED.  ie  they must be of the form drive:\path\file.ext
If they are not, then the command executed will be relative to the current
disk/directory.

Virtualization All Change!

Sun, 04 Mar 2012 17:14:52 -0500

Just two weeks ago, I revisited my virtualization options with a view to making the system more reliable - primarily by using mirrored disks.

In the end I stuck with a kludged up process for Citrix XenServer, but with a worry about how this would impact patching and upgrades.

This week my XenCenter instance told me that 6.0.2 was out and I should upgrade.

Now there are two ways to upgrade a XenServer; one is via the XenCenter console where it pushes the updates, the other is to boot off the CD and upgrade.

Things started to look worrying when the XenCenter method (which I’ve never tried before) reported losing connection to the XenServer, and looking in the logs it reports that there is insufficient disk space to perform the upgrade. Huh. Things took a turn for the worse when the CD method refused to believe the old install even existed, and would only do a new install (which might have been due to the system not finding or activating the md devices, and due to the installer only looking at physical disks).

So my worries were definitely on firm ground; how do I upgrade?

Now this upgrade was merely a series of minor patches and changes that didn’t impact me. But, going forward, I would end up being stuck with an out-of-date platform if I wasn’t careful.

So XenServer was, pretty much, out of the game. I was going to be forced to use something else.

I looked back at my work, 2 weeks ago, and decided to give CentOS+KVM another shot. The main reason I’d dropped this was that serial console installs would not work.

This time I bit the bullet, and decided to build out a “templating” solution. I built one CentOS 5.7 install and one CentOS 6.2 install. I used RAW image files. So now I can use “losetup” and “kpartx” and then mount the images and “sanitise” the result (disable SELinux, remove firewalls, add my default user/passwd, remove log entries). The result is an OS template image which can be copied around as needed. I then worked out a simple XML file which can be used to generate new virsh definitions.

The result is a BUILD script and an TEMPLATE.xml file which can be used:

# ./BUILD RH62.x86_64.template.img newstest d6:05:8b:cd:37:c0
Copying /var/lib/libvirt/images/TEMPLATES/RH62.x86_64.template.img to /var/lib/libvirt/images/newstest.img
Defining VM
Domain newstest defined from /tmp/tmp.NaiOX4rJNe

# virsh start newstest
Domain newstest started

% virsh console newstest
Connected to domain newstest
Escape character is ^]

CentOS release 6.2 (Final)
Kernel 2.6.32-220.4.2.el6.x86_64 on an x86_64

newstest login:

EDIT 2018/01/13 - the original scripts I used are no longer available so I have removed them as links. I don’t use templates any more.

In this case I specified a MAC address so that this new machine matched the DHCP config I’d already setup; otherwise I let virsh pick a MAC address at random and the new host gets a random IP address from the pool.

Using this method I don’t need to worry too much about serial installs; I don’t even need to do an install for new VMs I’ve previously templated.

Now I built these machines out using a 4Gb disk which results in approx 3.2Gb root. The OS takes up 700Mb, maybe. That doesn’t sound like a lot, but the idea is that data volumes can be added extra to this. Thus the ADD_DISK script, which can be used to add vdb…vdk as datadisks. For these I’ll likely not partition them, and use as raw devices, so it’d be easy to resize as necessary.

newstest.ttyS0% df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/vda2              3107344    740068   2209428  26% /
tmpfs                   251356         0    251356   0% /dev/shm
/dev/vda1               495844     49071    421173  11% /boot
/dev/vdb              20158332    176196  18958136   1% /datadisk

So far I’ve spoken about Linux. I also did a standard install of Windows to verify it works. Just as expected. I’m not sure if this is slower or faster than XenServer; it’s hard to tell when the display technology is different!

We’ll have to see how this works out, long term.

Virtualization Update

Mon, 20 Feb 2012 15:03:25 -0500

Two years ago I looked at some options for doing virtualisation at home. I decided on running Citrix XenServer. This has actually worked out quite well. So much so that I want to move some of my remaining physical hardware onto virtual.

And here I run into a problem. XenServer doesn’t want to work nicely with mirrored disks. It’s expecting SAN or similar to provide the redundancy for disks. Now people have worked out options to convert a XenServer to a RAID disk, but I’m very very worried about how upgrades might break the OS partition. The XenServer upgrade process appears to recreate sda and I’m afraid of it either not working or breaking the mirror. Of course I can’t tell until I actually have to do an upgrade, but I’m not sure I want to to risk it.

So… let’s see how the world has moved on in the past two years.

Now XenServer can “export” backups of VMs in “xva” format. This is a proprietary format that’s mostly a tar file with disk blocks. Fortunately you can convert these files into raw disk images.

Whatever I replace the XenServer with should be able to handle these image files. I don’t expect the Linux images to be hard, but the Windows images… hmm.

CentOS 6.2 and Xen

I’m primarily a CentOS user, so my first thought would be using CentOS 6.2. Now the GUI installer for this handles RAID, after a fashion. I created a mirrored root, mirrored /boot and installed a minimal OS. After that completed I converted the rest of the disk (sda3/sdb3) into a mirror and LVMd it. So far, so normal.

Now I start to run into issues; RedHat uses KVM for virtualization, and not Xen. But there is a process. Basically, install a xen kernel, and replace libvirt with one that understands Xen. Not the easiest to do, but not hard and, in theory, a one-off. Although kernel and libvirt patching might become an issue down the line.

So, within an hour of rebuilding the machine I had a fully mirrored Xen system and had test installed a new Linux image using a logical volume for storage. So far so good. I then installed virt-manager on my Ubuntu desktop, and was able to use that to create and install an XP image using a datafile in /var/lib/libvirt/images. virt-manager is nowhere near as powerful as Citrix XenCenter, but it works for basic stuff. Hmm, the distinction between using libvirtd for management vs Xen’s own tools seems to be a little confusing.

But then a problem… closing down a VM caused the whole machine to crash. Not every time, and not consistently, but I got 2 kernel crashes in one day. Not good. Not suitable.

At least following that method, CentOS 6.2+Xen is not stable enough for me.

CentOS 6.2 + KVM

I possibly should have tried this one first time around, since KVM is the native solution in CentOS. An important question will be whether the KVM emulated hardware is sufficiently close to the Xen hardware for the Windows sessions to work in it (assuming I can convert the disks, of course) and how much work will be needed to convert the Linux images, but.. first steps first; is KVM usable?

This time around I install all three virtualization groups from the CentOS installer. Also, rather than using virt-manager from my Ubuntu desktop I would ssh into the VM host and use the virt-manager from there (so RedHat^WCentos tools all the way. Needed “dejavu-lgc-sans-fonts” installed for words to display properly!). Hmm, this appears to add some niceness in building out the network devices and the storage pools, so the ISO images can be seen. I don’t recall seeing those options in my previous test but I might have been blind!

Now here is where I might be stupid. Using virt-manager I did a basic CentOS install and it worked just fine… except the network didn’t come up inside the guest. This might be a RedHat/CentOS oddity with Network Manager (I’ve seen messages similar on the mailing lists); setting the device to not use NM and to dhcp boot worked just fine. So then I tried another install, but this time without any graphical console; using virt-manager I removed the VNC, the input device, the screen. So there was just a serial connection. It started up as normal…but then after selecting “install” from the CentOS menu it froze. Umm.

So I try it from the command line:

virt-install -n t2 --vcpus=1 -s 8 -r 512 -b br0 -l /OS_ISO_images/CentOS-6.2-x86_64-bin-DVD1.iso -f /var/lib/libvirt/images/t2.img --nographics

And I get exactly the same problem. If I don’t use “–nographics” then I get a good install (but with the network config issue), whereas if I do specify it then the install freezes.

Hmm.

Debian 6 with Xen 4

(rant: “amd64” doesn’t mean AMD in Debian land, apparently! Also the installer is crap; does some work and makes you think it’s gonna run on its own, but then asks more questions later. On the plus side, the loader is better than the text-based CentOS one and works nicely with the existing mirrors.)

There are a quite few web pages that explain how to install Xen on Debian. It looks pretty simple. So let’s give it a try!

After getting my base Debian install setup, I followed some of the setup on that page. Hmm, Debian Xen seems to be a little different to CentOS (and XenServer?). When a domain is shutdown it no longer appears in the ‘xm list’ output (so you can’t just do “xm start name”; you have to do “xm create /etc/xen/name.cfg”).

The “xen-create-image” command is funky, and a great way of creating new Debian images. If I wanted a Debian image!

Let’s see if I can boot one of the older CentOS images I created in an earlier test… well, by setting the mac address and using pygrub then it worked to boot that.

bootloader = '/usr/lib/xen-default/bin/pygrub'
vcpus       = '1'
memory      = '512'
root        = '/dev/xvda1 ro'
disk        = [
                  'file:/VMs/t1.img,xvda,w',
              ]
name        = 't1old'
dhcp        = 'dhcp'
vif         = [ 'mac=52:54:00:01:72:1B' ]
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

But what about an XP VM? Looking at an example page, this seems to need xen-qemu-dm to be installed, and this drags in a lot of dependencies. But OK… let’s go for it!

# dd if=/dev/zero of=/VMs/xp.img bs=1 count=0 seek=8G
# cat /etc/xen/xp.cfg
import os, re
arch = os.uname()[4]
kernel = "/usr/lib/xen-default/boot/hvmloader"
builder='hvm'

memory = 512
shadow_memory = 8
name = "xp"
# Setting ne2k_pci is essential if you want the virtual NIC to work
vif = [ 'type=ioemu, bridge=eth0, model=ne2k_pci' ]

disk = [ 'file:/VMs/xp.img,xvda,w', 'file:/OS_ISO_images/XPSP3.iso,xvdc:cdrom,r' ]
device_model = '/usr/lib/xen-default/bin/qemu-dm'
boot="cd"

sdl=0
# Enable VNC
vnc=1
vncconsole=1
listen='0.0.0.0'
vncviewer=0
nographic=0
stdvga=0
serial='pty'
audio=0
localtime=1
acpi=0
apic=0

# xm create --vncviewer xp.cfg

Well, the installer kicks off, lets me format the disk, copies the files, then reboots. At this point I need to “xm vnc xp” to reconnect to the machine and the next part of the installer is running. Just as expected. Indeed, the build completed and runs just fine. Although the tightvnc viewer that comes with Debian doesn’t seem to work as well as the viewer with CentOS; definite mouse-coordination issues.

Let’s try the VM I created on CentOS.. hmm failed to start (it crashed and on restart brings up the ‘fail’ menu, but then fails again, even in safe mode). Hmm, OK, turning on acpi and apic and now the image boots! Neat. But, again, mouse coordination issues. Let’s try an image all the way from the original XenServer. Hmm… it’s running, but taking forever to get through the startup screen. Hmm. “xm list” always shows it in blocked state, but the indicator bar on the boot screen is moving. Hmm.

Hmm, after 11 hours it’s still in the same state. We might be able to kludge things, but… hmm.

However, I’m not feeling “comfortable”. And here we come to the core of the matter; the Xen install on Debian appears to be nice and stable. But it’s not very friendly. It’s all command-line, and (from what I’ve read), you need to have different config files for install vs run. Which seems odd. The CentOS libvirt method is a lot smoother and prettier… but doesn’t appear to be stable.

I’m not sure either is suitable to replace the XenServer. If the CentOS one hadn’t crashed (twice) in testing, then I might have continued further with it, but maybe not (considering the libvirt compatibility thing).

So this leads to the next test…

Debian 6 with Xen 4 and libvirt

Starting from a fresh install of Debian, we’ll do an install of virt-manager and see how well that improves the user experience.

apt-get install virt-manager xen-qemu-dm xen-linux-system

This seems to pull in world+dog, but that’s OK :-)

0 upgraded, 220 newly installed, 0 to remove and 0 not upgraded.
Need to get 177 MB of archives.
After this operation, 474 MB of additional disk space will be used.

Hmm, xend needed to be configured to listen to http requests on localhost:8000 before virt-manager could talk, but that’s a minor issue.

Let’s see what we can do with this version! OK, we can set up storage pools, so I can find the ISO images. But I can’t set up network bridging via virt-manager. Maybe that’s not important; let’s create a VM and see what happens! OK, the “create a VM” has more options for different types of Linux. On the final screen I manually specify “eth0” as the bridge device.

And the build fails:

xend.err "Error creating domain: device model '/usr/lib64/xen/bin/qemu-dm' not found"

That’s because it’s in /usr/lib/xen-default/bin/qemu-dm (/usr/lib64 is a symlink to /usr/lib). Hmm. OK, let’s symlink /usr/lib/xen -> xen-default and try again! Now I get the vnc based boot screen. Of course I only gave 512Mb to the VM and that’s, apparently, not enough for a GUI install, but the TUI install ran. So far, so good. Except that on reboot it needed me to kludge the settings to boot into HD. Hmm. The resulting image also has the network-manager problem seen previously; I’m beginning to think this is a CentOS/RedHat issue. Heh.

Oh, hold on… that’s doing a full virtualisation and not paravirt. Why? Oh, huh… “only URL or import installs work with paravirt”. Hmm, seems an odd restriction, but OK… I can do that. Umm, OK, not via PXEboot, but via HTTP it allows it. OK, I can do that as well. Here we go, a second build using paravirt. I do not recall if virt-manager on CentOS had that restriction or not! I don’t think it did; I’m pretty sure I select Xen (paravirt), but I might be misremembering.

Hmm, if I try to install a second image then virt-manager keeps grey-ing out (“not responding” type indication in Ubuntu).

Interesting… the second image booted cleanly and brought up the network! Hmm.

Let’s import that XP VM I just built in the previous test. Yup, that worked. And the virt-manager display keeps the mouse in sync properly. Although screen updates are a little slow and jerky.

Let’s try with the VM that still didn’t work after 12 hours :-) Oh, huh, in virt-manager I clearly get the choice of i686 or x86_64; I wonder if the previous import was the wrong CPU type. Hmm. No, it still does the same thing, irregardless.

Results

And, to be honest, I’m still not happy.

Modern OS management of Xen has definitely improved a lot over 2 years ago. But I don’t get the feeling of “comfort” that I get from XenServer. For the time being, I’m gonna stick with that and the kludged up mirror solution, and hope that the update process doesn’t break things!

… unless someone can point me at a better VM management solution!

The Windows 7 taskbar

Fri, 21 Oct 2011 12:23:22 -0500

People keep telling me how Windows 7 is so much better than XP. Eventually, at work, I get forced into using it. The first thing I notice is that the taskbar is now all icons, which you have to mouse over to see what windows each application has. And there’s no quick launch area any more; you can pin applications to the taskbar so they’re there.

However, I like having the XP option of having a program bar for each running program. Fortunately you can tell Win7 to do this. Unfortunately it’s not very friendly.

So here we have a normal setup; I have open Outlook, a message I’m writting, firefox and IE8. Note the position of the programs on the task bar.

Next, I opened OCS and Paint. Note that the programs have moved to the right ‘cos OCS has expanded from an icon to a program.

Finally, I opened a chat and started a new email. The outlook program item has moved to where mozilla was in the 1st screen shot, and the IE8 program has moved so far along the task bar that it’s moved to the second row! Also note that the pinned icons for programs not running are also moving around.

This is a real PITA ‘cos it means you have to continually scan the task bar to find the program you want to select. I can’t quickly move my mouse and click on “Word”; I have to find it, first!

With XP and the “quick launch” toolbar meant that programs didn’t bounce around on the taskbar in this manner; I could quickly and easily find quick launch icons, and long running programs gravitated to the left of the program area.

So it seems I have the option of losing information (glancing at the taskbar to see what is open) or losing ease of use (I have to hunt for the icon or program I want to use).

Sigh

A POTENTIAL FIX! After this rant, someone pointed me to these instructions. It’s a way of adding the quick launch toolbar back. And this solves 90% of my problems.

Basically the steps are:

Unlock taskbar (rightclick on it; ensure “lock the taskbar” is not clicked)
Add new toolbar (rightclick on taskbar; toolbars/New toolbar)
Select %appdata%\Microsoft\Internet Explorer\Quick Launch
This will add it to the right; drag it to the left, and resize it
Now remove the title/text options; right click near the dotted line and a menu should popup with “show text” and “show title” options; untick them *relock the taskback

You now have quick launch back!

IP6 Updates

Mon, 19 Sep 2011 23:05:47 -0400

Since those experiments, linode is now also providing native IPv6 so my linode was switched to the auto-provided address they provide. By default they only provide 1 IPv6 address but they allow rDNS to work, so I haven’t needed any more, yet!

IPv6 speeds internationally are a lot faster as well. I did some speed tests to a site in a UK exchange (connected at 100Mbit/s). It would saturate my home FIOS connection, peaking for periods of time at 3.65MB/s. When I attempted the same speedtest from my linode it was quite variable and bounced between 1 and 4MB/s, and was averaging slower than my FIOS. But when I tried it via IPv6 from the linode the average was 7.42MB/s with peaks up to 9.8MB/s. So here we see the IPv6 connection running over 3 times faster than the IPv4 connection! Funky stuff.

My home tunnel is now endpointed on a Seagate Dockstar, which I’ve got running Debian. The Dockstar is a “pogo-plug” type device, and has a Gbit network port. I’m not sure how fast the machine can drive it, but it’s definitely fast enough for IPv6 traffic to the outside world. This machine runs the tunnel and ip6tables. Now what this also means is that my linode and v-colo can talk “directly” to my home machines. So I no longer need my UUCP over SSL method of sending email; my home machine can talk SMTP over IPv6 to my virtual hosts (redundancy!) and those hosts and send mail directly to me. Faster, more efficient, more resillient.

What I do need to work out, however, is “fixing” the tunnel if my IPv4 address changes. HE tunnelbroker provide a web page to allow automatic updates, but they don’t like you hitting it for no good reason. And I can’t find an automatic way of reading what the IPv4 endpoint address is. So once or twice I’ve had my IPv4 change and so the tunnel freezes. I need to work on fixing that!

IPv6 on the LAN

Mon, 19 Sep 2011 22:54:15 -0400

Just for the lulz I fired up a CentOS VM (I knew that Citrix XenServer would come in handy!) and configured a tunnel to HE via that.

I then configured a static IP6 address on eth0 and fired up radvd. My main Linux machine automatically picked up an address on that subnet and could ping6 to the outside world. I could also ssh from my linode directly to my main Linux machine (did I mention I need an IPv6 firewall?).

Heh, the Mac Mini also automatically picked up an address as well.

So it kinda just works.

The problem I have, though, is that the IPv6 addresses being assigned are based on MAC addresses but I have no easy association between that and the host. So ipv6 assigned via autoconf and router discovery may be fine for dynamic hosts, but not so much for servers. And for dynamic hosts there’s not an easy association between the IPv4 and IPv6 addresses (eg dhcp214.home on IP4 is known; what’s the equivalent IPv6 address?). So it looks like I need a DHCPv6 server instead, so give out known IPv6 address ranges, rather than allowing autoconf via router discovery.

Messing around with ipv6

Mon, 19 Sep 2011 22:51:58 -0400

My v-colo at Panix can be configured to use IPv6. Now it looks a little bit like Panix is kludging routing slightly (they give you a /96 but with a /64 netmask). It’s very possible that their router is just at the end of a HE tunnelbroker, since HE are an upstream provider to Panix.

Enabling IPv6 on the v-colo was simple; I just enabled it on their “config” website and…it worked. I could even set up a rDNS entry for that IP. Since I don’t currently need more than the one address I didn’t bother looking at the other addresses.

My linode required a specific tunnel being set up. So I signed up for one, associated my linode IPv4 address with it. Then configured my CentOS machine:

Add to /etc/sysconfig/network

NETWORKING_IPV6=yes
IPV6_DEFAULTDEV=sit1

Create /etc/sysconfig/network-scripts/ifcfg-sit1

DEVICE=sit1
BOOTPROTO=none
ONBOOT=yes
IPV6INIT=yes
IPV6TUNNELIPV4=<HE's IP4 tunnel address>
IPV6ADDR=<Assigned IPv6 tunnel address>

Then “ifup sit1” and… it was on the network!

But, here, the rDNS of the tunnel is specific directly by HE. So I had to use one of the /64 addresses assigned to me. This was as simple as adding a line such as

IPV6ADDR_SECONDARIES=<one_of_my_ip_addresses>

“ifdown sit1” then “ifup sit1” and the address was assigned.

Now one thing I don’t understand is how the kernel decides on what source address should be used when making a connection, since “ip addr” shows both as a /64. But it seems to work; outgoing ssh connections show the connection coming from my routed address and not the tunnel address.

Of course, using a routed address meant I needed to setup rDNS for that. HE allows delegation of the routed subnet and a little bit of trail-and-error seems to have got that to work.

Because this is a test, I’m using a different name for my IPv6 addresses. When I’m convinced it all works then I’ll probably switch to having the same name for IPv4 and IPv6. Although I noticed my web server config doesn’t listen on IPv6. Umm.

Why am I doing this? I don’t believe IPv4 is going away for a fair few years yet (those areas with IPv6 only will have ISP provided proxies for web servers and 6to4 NAT gateway, for example) but I thought it’d be something to play with.

What’s interesting is this…

From my linode to my v-colo

IPv4:
6 packets transmitted, 6 received, 0% packet loss, time 5007ms
rtt min/avg/max/mdev = 83.076/83.313/83.760/0.436 ms, pipe 2

IPv6:
6 packets transmitted, 6 received, 0% packet loss, time 5001ms
rtt min/avg/max/mdev = 71.254/72.490/76.062/1.670 ms, pipe 2

Various traceroutes later, it seems the cross-country transit from NYC to LAX (which is HE, both ways) is quicker on IPv6 which surprises me.

I guess my next step will be to add a tunnelbroker connection to my home network. But that will require a little more thought; I don’t necessarily want to expose my Windows machines to the outside world (the NAT gateway acts as a nice little firewall for IPv4) so I’ll need to be a little bit cleverer. It also seems my preferred WTR54G firmware (Tomato) doesn’t support IPv6 so I’ll need to work on that as well.

*UPDATE:*

I realised that the way I was seeing traceroutes to my routed IPv6 range meant that it wasn’t actually using the end-tunnel IPv6 address for the route. So I just tried assigning one of the routed addresses directly to the interface and not use the tunnel address. It seems to work just as well, so I don’t have to worry about what source ip6 address will be used for outgoing connections.

Kerberos and IPv6

Sat, 09 Jul 2011 12:39:58 -0400

Not only have I been playing with Kerberos, but I’ve also been playing with IPv6. So, naturally, kerberos over IPv6 was a test I had to do.

Now because I’m only playing with IPv6 I’ve been using different DNS names; so kdc.spuddy.org is on IPv4 but kdc.ip6.spuddy.org is on IPv6.

So, test!

$ telnet -a -f kdc.ip6.spuddy.org
Trying 2001:470:1f07:dc4:3c46:1aff:fef4:d7a3...
Connected to kdc.ip6.spuddy.org (2001:470:1f07:dc4:3c46:1aff:fef4:d7a3).
Escape character is '^]'.
[ Kerberos V5 accepts you as ``sweh@SPUDDY.ORG'' ]
[ Kerberos V5 accepted forwarded credentials ]
Last login: Sat Jul  9 07:48:04 from kclient.spuddy.org
[sweh@kdc ~]$

Perfect!

But SSH…

[sweh@kclient ~]$ ssh -o 'GSSAPIDelegateCredentials=yes' kdc.ip6.spuddy.org
sweh@kdc.ip6.spuddy.org's password:

Ugh. It’s asking for a password, so the ticket wasn’t accepted. Ugh.

To debug it, I ran “sshd -d -p 2222” and tried connecting to that. This showed in the debug output…

debug1: userauth-request for user sweh service ssh-connection method gssapi-with-mic
debug1: attempt 1 failures 1
Postponed gssapi-with-mic for sweh from 2001:470:1f07:dc4:1c79:45ff:fe48:1534 port 58243 ssh2
debug1: Unspecified GSS failure.  Minor code may provide more information
Wrong principal in request

Wrong principal. Doesn’t sound good! I’d already ensured that the keytab had both ip4 and ip6 principals in it (“klist -k”).

Googling around makes me think this is a problem with older versions of OpenSSH. eg http://mailman.mit.edu/pipermail/kerberos/2007-January/011126.html

But now it appears the solution is upgrading to OpenSSH 4.5p1 along with the associated GSSAPI Key Exchange patch. The 4.5p1 GSSKEX patch supports a new config option (introduced in 4.4p1) “GSSAPIStrictAcceptorCheck=no” which, according to the man page:

If ‘no’ then the client may authenticate against any service key stored in the machine’s default store. This facility is provided to assist with operation on multi home machines

Huh, so if I make /etc/hosts on kclient have the IPv6 address associated with the kdc… it worked! So it seems the name the ssh client tries to connect to impacts whether the remote sshd accepts the ticket or not. Odd. This’d impact anyone with CNAMEs or multiple A records or… The “acceptor check” flag lets sshd check against any principal in the keytab, not just the first.

Of course, RedHat is “long term release” and it doesn’t look like this enhancement has been backported to RHEL5.

A little annoying!

Kerberos and Active Directory

Fri, 08 Jul 2011 22:52:08 -0400

So I built a quick AD domain based on W2k3 R2. I created TESTDOM.AD.SPUDDY.ORG as my AD domain, and made my primary DNS delegate that part of DNS to the AD server.

I was able to join an XP client to the domain.

So far, so good!

So then I built a CentOS 5.6 machine and configured it for Kerberos. set up krb5.conf:

[libdefaults]
 default_realm = TESTDOM.AD.SPUDDY.ORG
 dns_lookup_realm = true
 dns_lookup_kdc = true
 ticket_lifetime = 24h
 forwardable = yes

[realms]
 # TESTDOM.AD.SPUDDY.ORG ={
  # kdc = adprimary.testdom.ad.spuddy.org:88
  # admin_server = adprimary.testdom.ad.spuddy.org:749
  # default_domain = testdom.ad.spuddy.org
 # }

[domain_realm]
 .testdom.ad.spuddy.org = TESTDOM.AD.SPUDDY.ORG
 testdom.ad.spuddy.org = TESTDOM.AD.SPUDDY.ORG

I configured sshd for kerberos, same as for my real kerberos machine.

So far, so good:

sweh@dhcp219's password: 
Last login: Sun Jul  3 18:26:12 2011 from mercury.spuddy.org
$ klist
Ticket cache: FILE:/tmp/krb5cc_500_JYPNYK7309
Default principal: sweh@TESTDOM.AD.SPUDDY.ORG

Valid starting     Expires            Service principal
07/03/11 18:42:31  07/04/11 04:42:32  krbtgt/TESTDOM.AD.SPUDDY.ORG@TESTDOM.AD.SPUDDY.ORG
        renew until 07/04/11 18:42:31


Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached
$

Of course I’m using local users, but with Kerberos password authentication, and happily getting tickets.

Next step is to join the machine to the domain so I can use tickets. Here things get a little interesting. Samba’s “net ads join” command can seem to do it, but I didn’t want to modify the standard Samba config. Fortunately you can specify “-s” to a different config file:

# cat smb.conf.AD 
[global]
workgroup = TESTDOM
realm = TESTDOM.AD.SPUDDY.ORG
security = ads
password server = adprimary.testdom.ad.spuddy.org
use kerberos keytab = true

# net ads join -s smb.conf.AD  -U administrator
administrator's password: 
Using short domain name -- TESTDOM
DNS update failed!
Joined 'DHCP219' to realm 'TESTDOM.AD.SPUDDY.ORG'

The DNS failure doesn’t seem important; it’s probably an artifact of my having a different DNS domain for my machines than the AD server. But looking at AD from the W2K3 GUI, I see the host is created. So not we can create they keytab:

# net ads keytab create -s smb.conf.AD -U administrator
administrator's password: 
# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   5 host/dhcp219.spuddy.org@TESTDOM.AD.SPUDDY.ORG
   5 host/dhcp219.spuddy.org@TESTDOM.AD.SPUDDY.ORG
   5 host/dhcp219.spuddy.org@TESTDOM.AD.SPUDDY.ORG
   5 host/dhcp219@TESTDOM.AD.SPUDDY.ORG
   5 host/dhcp219@TESTDOM.AD.SPUDDY.ORG
   5 host/dhcp219@TESTDOM.AD.SPUDDY.ORG
   5 DHCP219$@TESTDOM.AD.SPUDDY.ORG
   5 DHCP219$@TESTDOM.AD.SPUDDY.ORG
   5 DHCP219$@TESTDOM.AD.SPUDDY.ORG

So far so good… I can login using tickets!

$ klist
Ticket cache: FILE:/tmp/krb5cc_500_JYPNYK7309
Default principal: sweh@TESTDOM.AD.SPUDDY.ORG

Valid starting     Expires            Service principal
07/03/11 18:42:31  07/04/11 04:42:32  krbtgt/TESTDOM.AD.SPUDDY.ORG@TESTDOM.AD.SPUDDY.ORG
        renew until 07/04/11 18:42:31


Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached
$ ssh -o 'GSSAPIDelegateCredentials=yes' dhcp219
Last login: Sun Jul  3 18:42:31 2011 from mercury.spuddy.org
$

At first attempt ksu didn’t work either.

But adding this domain_realm mapping:

  .spuddy.org = TESTDOM.AD.SPUDDY.ORG

and now ksu works:

$ klist
Ticket cache: FILE:/tmp/krb5cc_500_Zvavdv8343
Default principal: sweh@TESTDOM.AD.SPUDDY.ORG

Valid starting     Expires            Service principal
07/03/11 23:55:16  07/04/11 09:55:17  krbtgt/TESTDOM.AD.SPUDDY.ORG@TESTDOM.AD.SPUDDY.ORG
        renew until 07/04/11 23:55:16


Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached
$ ksu
Authenticated sweh@TESTDOM.AD.SPUDDY.ORG
Account root: authorization for sweh@TESTDOM.AD.SPUDDY.ORG successful
Changing uid to root (0)
[root@dhcp219 sweh]#

I guess I don’t understand the domain_realm mappings; I’d been cargo-culting. I guess they’re host DNS domain maps to kerberos realm. Which, really, doesn’t strike me as necessarily sane. Why wouldn’t the default_realm value from the [libdefaults] section take precedence, anyway?

It’s not all sunshine and roses, though. /var/cache/samba/gencache.tdb is modified; I don’t know if this would impact existing samba usage.

Beginning kerberos

Fri, 08 Jul 2011 22:52:01 -0400

Disclaimer: I know nothing about Kerberos. I’m learning it all from scratch.

I wanted to do some playing around with Kerberos (once I know Kerberos then I can look better at how to integrate with AD), so at home I set up a couple of CentOS 5.6 server VMs on my home network, built one out as a KDC (“yum install krb5-server”) and one as a Kerberos client talking to the KDC (krb5-workstation installed by default). This is a pretty good “getting started” guide.

On a machine that isn’t kerberized, I have no tickets

non-kerb-client% klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_500)

Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached

Note that the default filenames used are /tmp/krb5cc_ (for kerberos5) and /tmp/tkt for Kerberos4.

On the Kerberos client machine I configured sshd to perform kerberos authentication. So now I login to it:

non-kerb-client% ssh mylogin@kclient
mylogin@kclient's password:
Last login: Wed Jun 29 09:47:29 2011 from mercury.spuddy.org
[mylogin@kclient ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500_KhWkrB4506
Default principal: mylogin@SPUDDY.ORG

Valid starting Expires Service principal
06/29/11 09:48:53 06/30/11 09:48:53 krbtgt/SPUDDY.ORG@SPUDDY.ORG

Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached

Note the krb5 ticket file has a different name; this is a per-session kerberos ticket file that was created by ssh when I logged in.

I have a “krbtgt” which is a kerberos ticket granting ticket, which means I can now use this session to access other resources…

Let’s access another machine by ssh. In this case the kerberos key distribution center (kdc)

[mylogin@kclient ~]$ ssh kdc
Last login: Wed Jun 29 09:48:11 2011 from kclient.spuddy.org
[mylogin@kdc ~]$ klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_500)

Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached

Well, it let me in without needing a password… but my ticket didn’t carry over! That’s because I didn’t ask for the ticket to be forwarded…

[mylogin@kdc ~]$ logout
Connection to kdc closed.
[mylogin@kclient ~]$ ssh -o 'GSSAPIDelegateCredentials=yes' kdc
Last login: Wed Jun 29 09:49:11 2011 from kclient.spuddy.org
[mylogin@kdc ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500_GSTIKd5634
Default principal: mylogin@SPUDDY.ORG

Valid starting Expires Service principal
06/29/11 09:49:21 06/30/11 09:48:53 krbtgt/SPUDDY.ORG@SPUDDY.ORG

Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached

Ah ha! Here we go. By telling ssh to delegate credentials, we get a new TGT on the remote server, which can be used for further access.

ktelnet can also do this…

[mylogin@kdc ~]$ logout
Connection to kdc closed.
[mylogin@kclient ~]$ telnet -a -f kdc
Trying 10.0.0.23...
Connected to kdc.spuddy.org (10.0.0.23).
Escape character is '^]'.
[ Kerberos V5 accepts you as ``mylogin@SPUDDY.ORG'' ]
[ Kerberos V5 accepted forwarded credentials ]
Last login: Wed Jun 29 09:49:21 from kclient.spuddy.org
[mylogin@kdc ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_p5780
Default principal: mylogin@SPUDDY.ORG

Valid starting Expires Service principal
06/29/11 09:51:41 06/30/11 09:48:53 krbtgt/SPUDDY.ORG@SPUDDY.ORG

Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached
[mylogin@kdc ~]$ logout
Connection closed by foreign host.

Different temporary filename, but it works the same.

Kerberos can also be used for ‘su’

[mylogin@kclient ~]$ ksu
Authenticated mylogin@SPUDDY.ORG
Account root: authorization for mylogin@SPUDDY.ORG
Changing uid to root (0)
[root@kclient mylogin]# klist
Ticket cache: FILE:/tmp/krb5cc_0.1
Default principal: mylogin@SPUDDY.ORG

Valid starting Expires Service principal
06/29/11 09:48:53 06/30/11 09:48:53 krbtgt/SPUDDY.ORG@SPUDDY.ORG
06/29/11 09:51:06 06/30/11 09:48:53 host/kclient.spuddy.org@SPUDDY.ORG

Kerberos 4 ticket cache: /tmp/tkt0
klist: You have no tickets cached

[root@kclient mylogin]# cat /root/.k5login
mylogin@SPUDDY.ORG

I’m pretty sure this is not a good idea; at work we want stronger controls over who can access root, so it means we’ll have to check for .k5login files if we ever switch to kerberos for authentication!

Note that ‘root’ now also has the host key in the kerberos cache.

Note that Kerberos is just an authentication system; it doesn’t handle naming services nor authorizations; it’s just a “secure password repository”.

This is exactly the sort of testing I wanted my XenServer for!

X-No-Archive considered pointless

Tue, 05 Jul 2011 22:12:46 -0400

Because of who we are and what we do, a number of people who post to newsgroups like to hide themselves; use anonymous names, try to stop archives from retaining messages and so on. I, personally, am open with who I am and what I am but this doesn’t mean I decry the attempts of others to maintain anonymity.

One such attempt is the use of X-No-Archive: headers in newsgroup postings. This tells google to not archive the message. Other archivers may also use this convention.

Note an important thing about this though: the line must either be a header or the first line of the message.

This has a major implications on follow ups.

By default, a followup to a X-No-Archive message is not itself X-No-Archived.

The new message will not have the message header in place (if the original had it in the header) and any message body with the line in would be quoted (typically with a > symbol) and appear after attributions line (eg “Fred Bloggs wrote:“).

For convenience, I’ll simply refer to this header as XNA in the rest of this document.

I have been taken to task, before, for not XNAing my responses to a XNAd message. It was even, once, criticised as being deliberate. As can be seen from above, a followup to an XNA message is not, itself, XNAd without additional action on my behalf. And that action is actually quite an onus:

I don’t automatically put XNA headers in my post because I don’t want my articles non-archived (tin allows per-newsgroup headers if needed)
If a message has XNA header then I don’t even see them becuase I have my newsreader set to hide them. Most definitely X- headers (a “do what you like” header with no standing in RFC-1036 beyond that) don’t get copied into reply messages (otherwise X-Face: and other stuff would also get copied)
If the XNA line is the first line of the message then when I quote the response then the quoted line (a) has a > in front and (b) isn’t the first line of text, and so is ignored by google.

This means that if people want me to honor the request to propogate I have to

notice it
go out of my way to ADD text to my message
go against my desire to want to archive my text

Basically, XNA has no standing in Usenet; it’s a flag for google (and other archivers that may decide to follow the standard). It’s no better than X-No-Ahbou: or any other of the potential 1000 X-No- headers that people may decide to create off the top of their head (X-No-War-In-Iraq: perhaps?)

Now, it’s been argued that people can have good reasons why they don’t want their posts to be available forever and that by failing to honour the XNA request is in the same general category as outing a person (but not as bad); outing as in identifying them as a member of some minority that may cause them distress elsewhere (eg a closetted gay person, a BDSM practioner, whatever).

I disagree totally. The only place XNA has any meaning is in the google groups archive (and potentially other archives which also use this convention). I effectively have an archive; my own personal newsserver never expires messages and so has messages going back to Jun 6 2001. It most surely never follows any X- header request because it’s a Usenet server and X- headers are meaningless to Usenet. I can (and have) quoted messages from the past when they are relevant.

Contrarily, I consider if offensive that other people wish to prescribe and force behaviour on me that I do not wish, especially when their are no RFCs or BCPs that cover the situation, and (indeed) where the RFCs say that the header in question has no meaning.

RFC-822 (the one that defines headers and which RFC-1036 - usenet - refers to) says:

    extension-field =
                    <Any field which is defined in a document
                    published as a formal extension to this
                    specification; none will have names beginning
                    with the string "X-">

     user-defined-field =
                    <Any field which has not been defined
                    in this specification or published as an
                    extension to this specification; names for
                    such fields must be unique and may be
                    pre-empted by published extensions>

That means NO field defined by any RFC standard will begin with X- which implies that the XNA header is a user-defined-field.

Another argument could be that this is a fairly well defined convention and that while the standards don’t specify, there is no reason to not accept this. My response is simply that HTML format messages and MIME encoded binaries are also well defined conventions. If it becomes the onus on the person following up to check for and maintain a non-standard header because of the wishes of the sender, then why shouldn’t the sender also be allowed to send HTML messages, with the onus on the reader to get software that can handle it?

This is why we have standards documents.

What this boils down to is that the XNA concept is fatally flawed as a means of keeping responses to messages out of google. Responses (either by accident or by design or by simple not-caring) may easily quote a XNAd message and not have the header itself. In my case it’s simple “not caring”; if someone posts a message then they must expect it to be quoted and followed up, and along the way it may easily be archived.

Anyone who relies on XNA to be no more than a request for their own message is deluding themselves as to the state of Usenet.

As an alternative, people concerned about their anonymity could create a free webmail address (eg hotmail, excite, yahoo, whatever) and use that in their postings. If they XNA their own message, then the originating IP address (their ISP) won’t appear in the archives and any followup (be it XNAd or not) will merely quote their webmail address. It would take a deliberate act to copy any NNTP-Posting-Host headers into the follow up message (or an act of “pilot error” - ie a mistake). Their anonymity is maintained. If they don’t want people reading their words in 5 years time, then don’t post in the first place. Once a message has been thrown out to usenet it has an indefinite life time. XNA does not, and can not, change this.

Virtualization Options

Sun, 06 Jun 2010 16:52:02 -0400

Many of us are geeks who like to play with technology “because it is there”. We might want to try out a new OS, or a new piece of software. Maybe install a beta version of something, or be able to test a client-server setup. Historically that has meant having one (or more) test machines, configured as multi-boot. In 2002 I spent $600 on a Celeron 1200Mhz machine with 256Mb RAM and a 40Gb disk for precisely this purpose; it multi-booted into XP, NetBSD, Solaris 86, Fedora… at that point I ran out of primary boot partitions. sigh

Today, however, machines are a lot more powerful. Since I had a spare case, power supply and 3*500Gb disk sitting around I spent $500 getting a new motherboard ($90), an i5-750 CPU($200) and 8Gb of RAM ($210). I picked this setup because it was a VT-x compatible CPU so could do hardware virtualization for Windows guests. Also, in the worst case, it could quickly be used as a standby machine if my primary server (Q6600, 4Gb RAM) died. Redundancy :-)

Then I had to decide on what virtualization technology to use on this machine.

Types of hypervisor

Hypervisors come labelled in two types; “type 1” and “type 2”. It’s really a little but kludgy to have this separation but…

Type 1 hypervisors are typically small kernels that essentially act as a message bus between guest OS’s and a “control domain”. The guest may think it’s talking to a SCSI controller but really this talks to the hypervisor, which passes the request to the control domain, which then does the work of talking to the hardware. Hypervisors built on this model can be quite small and are potentially suitable for embedding into server firmware. Because of this they’re commonly known as “bare metal” hypervisors. This is what Sun have done with some of their newer hardware; the sun4v platform is a hypervisor’d virtual SPARC chip; LDOMs are guest domains. In the Intel (read “Intel, AMD and similar”) space Xen is the common OpenSource type 1 hypervisor; VMware ESX is a common commercial one. Microsoft’s HyperV is a Type 1 solution.

Type 2 hypervisors run a step away from the hardware; you have your primary OS install and the hypervisor runs inside that; this is called “hosted”. As with type 1, guests request access to their SCSI controller, the hypervisors traps this and then makes a request of the primary kernel. This sounds inefficient and would be, except that common type 2 hypervisors have kernel loadable modules and so essentially run in the kernel space of the primary OS. This may make them as efficient as Type 1. VMware Server, VMware Player, and VirtualBox are examples of Type2 hypervisors.

What about full machine virtualisation? Where the whole machine is “virtual”. I consider this an extreme example of a type 2 setup. An example of this is ‘User Mode Linux’, where the Linux kernel itself runs as an application instance inside another install.

Essentialy, with a type 1 hypervisor the hypervisor is loaded first and all the OS instances (control domain, guests) load afterwards. With a type 2 hypervisor an OS instance loads first, and the hypervisor afterwards (or, with kernel modules to confuse things, at the same time).

Paravirtualization? Huh?

In order for the guest to access things like network or disks they need to have drivers. So, typically, your VM solution will emulate some common hardware (maybe a RealTek or Intel ethernet card; an IDE or BusLogic SCSI disk). The guest will have drivers for this already, so they’ll “just work”. But as you can guess this may not be very efficient and may result in many context switches as the guest’s requests bounces around the system before it gets to the physical hardware. To help shortcut this the hypervisor may expose an easier-to-emulate interface to the guest which can be processed a lot quicker. To the guest these devices will look like a different piece of hardware entirely and so will require drivers. This technique is called paravirtualization and can result is massively improved performance in the guest.

Hardware assisted virtualization

A hard part of virtualization is emulating the CPU. Some of the instructions are, apparently, complicated and expensive to virtualize and this causes the hypervisor to do lots of work. Intel created something known as “VT” (aka ‘Vanderpool’); AMD have AMD-V (‘Pacifica’). This allows the CPU, itself, to virtualize some parts of the instruction set taking the load off the hypervisor, and speeding up virtualization.

Related to this is the ability to virtualize hardware devices, such as PCIe plugin cards. Intel call this VT-d (Virtualization Technology for Directed I/O). With this it’s possible for a guest OS to have direct exclusive access to hardware devices (maybe one of the USB controllers, or a disk controller). This is, really, a layer violation but it can be a performance gain or allow VMs to access hardware that the hypervisor can not emulate. My machine, apparently, supports VT-d in the BIOS but either the chipset (H55) doesn’t support it or something else is wrong; the capability isn’t available to the hypervisor.

What are my requirements?

I wanted a virtualization platform that would let me use the computer to host different OS images and run them. I wanted it to “just work”; even though I’m a geek and enjoy playing with technology this was one area where I wanted to play with the VMs, not with the VM technology. I didn’t want to delay messing around with <x> because I had to fight my VM system of choice in order to deploy new images.

Obviously stability was a requirement; if I had to keep rebooting the host because the platform then I’d get annoyed.

Got to be able to run Windows. If I want to test-drive Windows 2008 (well, I might on a trial license) or test W2K3 was an Active Directory controller or whatever… Hmm, could this be used to work around the expiration of the Windows trial license? Whenever the license is due to expire just destroy the VM and rebuild it. For something like AD have two servers, promote the other to primary, bring up a new replica. Heh. Not that I’d ever advocate working around license agreements like this!

No artificial limitation on the number of guest OSes; I don’t want to be stuck just running 4 VMs.

CLI to manage it would be nice; GUI to manage it almost essential (see ease of use, my primary requirement).

You’ll notice that speed wasn’t a primary goal; full virtualization or paravirtualization wasn’t too important, since these weren’t high throughput high demand systems. So what I’m testing for isn’t performance. This also isn’t a long term test. I’m primarily focusing on ease of use, of of setup, ease of administration. That is, after all, my primary goal.

So what did I test?

My primary server OS of choice is CentOS 5 (currently 5.4). Partly because I’ve been using RedHat variants for… 14 years? I started with RedHat 4.0 (no, not Enterprise Linux 4, just standard “RedHat”) around 1996, from an InfoMagic disk. Previously I’d used home-grown setups, Yggdrassil, SLS, MCC-Interim… Not sure when I first got hold of a Debian disk but by this point it was too late for me to change. It also helps that my employer’s primary Linux OS is RedHat so what I use at home is very close to what I use at work. RedHat/CentOS comes with two different virtualization technologies (Xen and KVM) so I tested both of them. I also tested the commercial version of Xen (Citrix XenServer). No comparison would be complete without VMware ESXi. I’ve previously used VMware Server and VirtualBox. And I daily use UserModeLinux.

So that’s the list:

RedHat (ahem, sorry, CentOS!) 5.4 64bit Xen
CentOS 5.4 64bit KVM
Citrix XenServer 5.5
VMware ESXi 4.0
VirtualBox 2.2
VMware Server (version unknown)
User Mode Linux (2.6.20.7 based kernel)

For the CentOS/Citrix/VMware solutions I let my existing machine act as a DHCP server, so the management interface was picked up from a static DHCP config.

CentOS 5.4 64bit Xen

Stick the DVD in the machine, boot off it, install as if you’re doing a normal Linux install, select the “virtualization” group. And let it go. 20 minutes later your machine will be ready with a CentOS Dom0 control domain and the Xen 3.1.2 hypervisor. Simple! Being Xen, there are plenty of command line tools available for the install, create, destruction of guest domains (DomU). Some of the jargon is a little quirky (for example “destroy” doesn’t mean destroy the VM image, just the running VM instance). But I didn’t want to learn the ins and outs of the command line. Fortunately CentOS comes with an X based GUI, “virt-manager”. This is pretty minimal, but sufficient to build an OS, manage resources, connect to the console… basic requirements.

First up, install a CentOS guest image. Simple process… I told it to bridge its ethernet device to my main network (so it looks like it’s on the LAN). Heh, the Dom0 is Linux so I NFS mounted my ISO images to this and now they’re available to act as boot CDs/DVDs for the build process. This is simple; under 10 minutes later I had a fully working CentOS guest image… and it was running paravirtualized as well! Neat! This found my DHCP server and got an address and did everything I expected of a newly built CentOS machine.

This is what I wanted from a virtual server; building a complete server in minutes without getting off my arse.

So, next, Windows. Follow a similar install path from the virt-manager GUI and… now things didn’t quite go according to plan. Just creating the raw disk image took a long time, then booting for the first time caused a hang. The guest instance just didn’t boot cleanly for installation. Hmm. Indeed it looks like the hypervisor started to slowly freeze up. The DomU became unresponsive. I couldn’t login on the console… needed a physical reboot. Ugh. But, OK, after the reboot I started the guest and the install carried on. The Windows setup process ran, copied data to the hard disk, then rebooted…. and here the Xen guest stopped. Needed to be started again and the OS install then carried on as normal and completed. To be fair, this step is documented. It just makes unattended installs harder. Not that the standard Windows install is anywhere near unattended anyway (asking questions 5 minutes into the process sigh) so it’s not a big loss. But, anyway, I had a Windows instance running. However I had stability concerns (would I need to reboot every time I created a new windows instance?). So I tried again… yup, reboot needed! Oh dear. I think this fails one of my primary requirements. It seems a fully patched CentOS 5 64bit Xen install isn’t sufficiently stable using full virtualization. This could easily be because it’s an old old version of Xen (3.1.2; current version is 3.4.5).

Data is stored in /var/lib, including the disk images. So either the relevant areas need to be mounted from a separate partition or a big root disk created. It is possible to store data elsewhere but SELinux may get in the way and need reconfiguring or turning off. Personally I turn it off.

CentOS 5.4 64bit KVM

Nicely enough, it’s possible to add KVM to my previous install. “yum install kvm” adds a new kernel and 3 other rpm’s. Reboot into the non-Xen kernel and we have a KVM system. Apparently this can also be done at initial install time by selecting the right options in the “virtualization” options menu. So an equally simple install compared to the Xen variant.

I’m not sure if this is considered a Type 1 or Type 2 virtualization. Not that it really matters. RedHat nicely hides the differences between Xen and KVM behind the same interfaces, so virt-manager is also used to manage KVM instances. This time I didn’t bother with a CentOS guest, I went straight to building Windows. And this went off without a hitch. A fully virtualized Windows machine.

Well, I say without a hitch… there was no option to select a bridge onto my LAN; it could only use a NAT’d network. Unlike the Xen install (which creates this bridge out of the box) the KVM install doesn’t. The RedHat documentation does explain how to build a bridge but it’s not automatic. The guide also describes to how convert this instance into a paravirtualized instance. Except it’s not quite so easy and requires manual hacking of config files. Yes, it does work. It’s definitely a step up from the Xen instance (it’s stable!), but I wasn’t feeling the “friendliness”.

Citrix XenServer 5.5

My hope, here, was to be able to install this onto the 2nd hard disk so it could co-exist with my physical Windows instance and the CentOS instance I’ve just installed. This’d give me a multi-boot test platform so I could switch and compare. Fortunately it allowed me to do exactly that! At install time a list of disks is presented, so I selected the 2nd one. A few minutes later we had a XenServer install.

The boot sequence looks familiar… huh, under the covers it’s actually CentOS 5.2 (as reported by “rpm -q centos-release”) with Xen 3.3.1. The console comes up with a text menu; let’s you select various statistics, start/stop VMs, reboot etc. It doesn’t let you create new VM instances. You can ssh into the dom0 and access the same menu remotely, so you don’t need to be at the console.

Out of the box you get a limited time license; to fully activate the service you need a free license from Citrix. A little annoying. More annoying is the fact that this license only lasts 1 year, so it has to be renewed. Let’s hope Citrix don’t change their policies and stop giving out free licenses, or use this to force people onto the upgrade treadmill.

To create VM instances requires a piece of software that only runs on Windows. A little annoying, since it just makes https webservices requests. Well, I guess it should also be possible to do the same via the dom0 command line. But see “ease of use”. For my purposes a Windows client it is. It’s pretty small and installs quickly. From here you can license your server (also see non-subtle adverts for “Xen Essentials”, a paid for enhanced interface) and perform various management functions, including create/delete/start/stop VMs. You can also attach to the console of the server and each VM.

Templated support for Guest OS’s is limited (RedHat and variants - Centos/Oracle/XenApp, Debian 4 and 5, SUSE server, Windows Variants). Installation of guests using the ISO library I’ve got was simple. My CentOS guest came up properly virtualised. I could then attach the xs-tools CD image (directly from the console tab, nice) and install the guest tools which allows the guest to better use the paravirt interfaces and report statistics to the manager. A Windows install was just as easy; install the OS, attach the CD. And the tools come with Windows paravirt drivers (at least for XP Pro) which just installed and worked. I like that.

But that’s pretty much where paravirt support ends; I test installed OpenSolaris 2009.06. I know this has paravirt support…but under XenServer it runs fully virtualized. I guess I could delve into the internals and work out the right kernel and stuff but…see ease of use. Similarly a Ubuntu Server 9.10 came up fully virtualized. XenServer really needs to support more OS platforms, both for paravirt and for xs-tools.

Stability appears to be rock solid.

A nice touch in “guest installation” manual is a description on how to convert a Windows build into a template (including using sysprep) so more instances can be cloned quickly. Given how slow a standard Windows build is, this is nice information.

It’s possible to create virtual networks totally internal to the server (no connection to the outside world) so you can build connections between servers. I couldn’t see an easy way of making the XenServer act as DHCP server for that network so either you’d need to run your own DHCP server or make sure all the machines were statically configured on that LAN.

For the Linux minded, the install takes the whole disk; partition 1 is the dom0 (CentOS 5.2) and takes 4Gb; partition 2 is of the same size; not sure what that is used for. The rest of the disk is assigned to a VG. Guest instances are LVs carved out of the VG. This is flexible (add a disk, extend the VG, more guest capability) and simple. You don’t need to know any of this, though, to use XenServer.

On the plus side I can also make this work from my primary grub partition with chainloader, so I can have a physical Windows instance for when I actually need one (game playing?) and boot into XenServer for virtualization and my test instances very easily.

Ease of install, ease of use, the simple Windows management GUI are all pluses. Yearly license and limited paravirt/xs-tools support are minuses.

VMware ESXi

After my experiences with XenServer I was hoping for something similar here. Obviously that didn’t happen. My hardware was considered “consumer grade” and VMware ESXi only supports server grade hardware. Essentially, check the hardware compatibility list carefully before installing ESX. Fortunately other people had worked out how to get ESXi to work on similar hardware, and along with some guesses of my own I managed to get it working. As with XenServer I could select a disk, so disk 3 went to ESXi. Install, reboot (more hacking needed here) and I had a running ESXi instance.

Not exactly a friendly out-of-box experience, but now I’ve documented it (and kept a copy of the required files) I could rebuild this pretty easily. I won’t hold it too badly against VMware :-)

ESXi comes with a 90 days trial license. You can get a free indefinite license by registering at VMware. Nice. Except functionality in this freeware is very limited. Here’s a comparison:

Trial license Product Features:
    Up to 8-way virtual SMP
    vCenter agent for ESX Server
    vStorage APIs
    VMsafe
    dvFilter
    VMware HA
    Hot-Pluggable virtual HW
    VMotion
    VMware FT
    Data Recovery
    vShield Zones
    VMware DRS
    Storage VMotion
    MPIO / Third-Party Multi-Pathing
    Distributed Virtual Switch
    Host profiles

The indefinite license has:

Product: ESXi 4 Single Server Licensed for 1 physical CPUs (1-6 cores per CPU)
Expires: Never

Product Features:
    Up to 256 GB of memory
    Up to 4-way virtual SMP

Pretty minimal set of features!

One thing VMware ESXi does support is “pass-through PCI”. Except my hardware doesn’t seem to support it.

ESXi normally hides the control domain; you can’t access it. Normally you’re meant to use their CLI emulation (perl wrappers calling webservices).

However if you switch to console 1 (ALT-F1) and type

unsupported

and then enter the root password you created then you get access to the command line. It’s possible, from here, to enable ssh. Although it’s a Linux based OS the software looks more like an embedded Linux system (busybox, ash, dropbear etc). Not that this has any impact on the usage of VMware.

As with XenServer, proper management of the server requires a Windows client. In fact XenServer is a very clear copy of VMware, from how the console looks to the layout of the Windows GUI. Except the VMware version is massively heavier. 330Mb for VMware (vs 15Mb for XenServer). The VMware GUI looks more “next generation” but this can mean “harder to find stuff”. But, yes, all the functionality you need is there to create and manage VMs.

All in all the experience was very similar to XenServer; not surprising if XenServer was designed to copy VMware look’n’feel. I found the VMware version slightly harder to use; some of the options were not in obvious places. A few times I remmeber thinking “I know I’ve seen this option… where was it?!“.

Again, virtual networks can be built; again I couldn’t see how to make ESXi act as a DHCP server to the network.

Neat idea; effectively there’s an app store available from the client where you can download and install prebuilt images into your server. Want to test Zimbra? Click-click-deploy… there’s a Zimbra instance on your network. The implementation wasn’t the best in the world, but the idea is good.

Windows paravirtualization just worked. But this time I kinda expected it. But the slowness of the GUI and lack of direct access to the control domain were beginning to annoy me. I understand VMware has more guest tools availability than XenServer, but I never tested them.

One downside to the Linux guest tools; they don’t appear to dynamically handle changes in kernel so you might find yourself running fully virtualised network/disk devices by mistake.

VirtualBox 2.2

I didn’t test his recently; this is from earlier usage (Jan->Jul 2009).

This is a type 2 hypervisor. I loaded it onto an existing CentOS 5 install. Because it requires kernel modules to work effeciently it needs to be aware of primary OS kernel updates; to handle this it uses DKMS. It works well and is, essentially, transparent. You can upgrade your kernel and on reboot the vbox drivers are recompiled to match.

This comes in 2 editions; VirtualBox and VirtualBox OpenSource (OSE). Since I wasn’t overly concerned about OpenSource purity (I wanted something that worked) I went for the closed source enhanced version. This adds built in RDP servers, ability to to pass USB ports from the host to the guest, and the ability to use USB ports over the RDP protocol (with a suitably enhanced rdesktop client, that they provide). Given one idea was to replace my sole Windows desktop with a Linux machine but I still needed Windows for ActiveSync to my cellphone, this looked like a nice idea…

Since I was running this on a Linux machine, the management GUIs were all X based. I understand it can also be run on a Windows host with Windows GUIs, but I never tested that.

Client installation was simple enough; Windows installed, ran, worked as expected. At the time I tested there were no paravirt drivers; today it seems there are some. The wonders of OpenSource; one person creates them, many projects take advantage. Similarly OpenSolaris loaded and installed just fine.

The main problem I had with VirtualBox was that it felt like a desktop virtualization environment. By default all the images lived in $HOME/.VirtualBox. This is great is you have many users who want to manage their own virtual environments; not so great if you want simple automation (‘start these VMs at boot time’). I wrote some simple rc scripts to work around this, but the resulting solution felt like a kludge. It also felt too dependent on my keeping the host OS stable; if I ever update from CentOS 5 to CentOS 6 (it’ll happen, eventually) or even switch to another Linux distro (maybe not) then I didn’t feel comfortable that VBox would easily migrate.

VMware Server (version unknown)

Another type 2 hypervisor; I use this (infrequently) at work to load up an older OS base install that I use to compile software; this install is minimally patched so I can be relatively sure that the software I build is compatible with all the deployed instances in the company; there shouldn’t be anything with a lower revision than this!

This doesn’t use DKMS, so if I change the kernel then I need to re-run a script to ensure the driver interfaces are recompiled and loaded. It warns you at boot time, but who reads boot messages? Really, VMware should look into DKMS for their tools (both server and guest). Otherwise there’s little to say about this… “it works”. It’s less desktop oriented (to my feeling) than VirtualBox, but pretty much requires root level access to do anything.

There’s a similar worry with Virtualbox about major OS upgrades.

User Mode Linux (2.6.20.7 based kernel)

This is the odd-one of the bunch. It’s not a virtual machine in the above context. What we have here is a copy of the Linux kernel where the devices are emulated. The whole Linux instance runs as an application inside the host. As you can image, this is hardly efficient. But it works. There’s no pretty GUI, there’s no automated deployment tools… there’s nothing that I said I wanted. So why am I looking at User Mode Linux (UML)?

6 years ago I got a Virtual Private Server (VPS) at a company called Linode. At the time they were using UML. Also at the time I was using an old Pentium Pro 200 as my firewall, and running Vserver to run security separated instances; I ran one for email (UUCP over SSL), and one for ssh and http. They were effectively bastion hosts. The core OS ran PPPoE and so was my internet gateway onto DSL. Time went forward and the PPPoE gateway got replaced with a Linksys WRT54G, but the PPro still lived on for the bastion hosts. This was lunacy. So I sat down and built out my own toolset to build and deploy UML instances. And, mostly, they work! I can deploy a new Linux instance in 3 minutes. Efficiency isn’t as bad as I expected either. I had a P4 3Ghz HT; I loaded a Postgres database and wrote code that took 2 hours to run (heavy transformation of tables). I created an equivalent UML instance on the Q6600, 2.4Ghz and the same code ran in 1.5 hours. Despite all the software layers behind UML it ran faster than the physical machine. Maybe the host OS disk caching helped (1GB RAM on the P4, vs 4Gb RAM on Q6600), but the Q6600 was doing more different work.

The neat thing about UML is that its footprint is so minimal; you can assign 128Mb of RAM and 300Mb of disk to an instance and you know that’s exactly what is taken up. This made it perfect for my bastion hosts.

Indeed I ported this technology to work and one of my DR instances is actually a UML instance running on my desktop. Sssh!

I’m a fan of UML, but it’s really not a general purpose virtualization solution. Apart from the fact it’s Linux only, you do need to be a heavy geek to use it.

Conclusion

RedHat Xen… too unstable. I can’t use it. Potentially I could use the latest OpenSource Xen drivers instead of the RedHat provided ones, but then I’d need to make sure the virt-manager stuff all worked properly and I’d be worried about OS updates blowing away customization. I don’t want to use a different Linux distro, because that’s yet another variation to support and manage.

RedHat KVM… technologically it may be there. User friendliness, umm. Maybe if I sat down and built out a toolkit for myself so I could quickly and easily deploy instances (shouldn’t really take more than a weekend of trial’n’error) then this may doable.

Citrix XenServer… best of the bunch so far; just wish it supported more platforms for paravirt and tools. And that one year license….

VMware ESX4i… I wanted this to win because it’d give me an idea of how we do stuff at work. But it was just a headache from start to finish.

VirtualBox… if you want virtualization on a desktop OS then this is a really strong contender. The USB enhancements make it stand out. I didn’t feel the love as a server based solution, though.

VMware Server. Strong basic virtualization. Again, good for your desktop OS, but if you want to use it as a server then you have the additional host OS patching requirements.

User Mode Linux. Not in the same class, but really good for small footprint minimal overhead low-power installs.

What am I going to use? I’m not sure! I’m tending towards Citrix XenServer. But not decided, yet.

Resources

Thin Client Options

Mon, 19 Apr 2010 19:21:59 -0400

In my spare room I have what I grandiously like to call a library. (By library I mean approx 112ft of bookshelf space, on 3 of the 4 walls). What does any library need? A computer! Internet access, ability to print, access files etc. I have a random vision of the future of having my eBooks managed “somehow” (plug the eBook reader in, download the book(s) I want…).

Nothing high powered; possibly playing youtube videos would be the hardest thing this computer would need to do. However I would like this machine to be able to print to a locally connected (USB) printer and also scan from it (it’s a HP printer/scanner). I could live without printouts (especially since that’s an inkjet printer so the ink’ll probably dry out before I use it again!) but the scanner is one of the reasons I want the machine there (eg to scan the covers of my books for my online index).

But first I need a computer. Most of the time this machine is going to be turned off (or, if power drain is sufficiently small, in standby mode) so a quick boot is required. Optimally no moving parts (no hard disk, no fans). It doesn’t need to be Windows (indeed, probably better off not being Windows, given the headache of keeping patches and antivirus software up to date!). So this sounded like a perfect job for a diskless X terminal. Netboot, NFS mount root, etc etc.

My first attempt at this was an iOpener; Panix used to use these when doing “internet rooms” at conventions; they’d hacked them so they had a NetBSD kernel and then would NFS mount everything else. Just needed the one server for the con room. These iOpeners didn’t have ethernet, but a USB dongle worked adequately for 10baseT. I’ve previously done a minor consulting favour for Panix and in lieu of any recompense I took two of these devices off their hands. I played around with them a little, but never felt the “love” or desire to actually get them working. It might be the small RAM (its been a while since I powered them up; maybe 64M?) or the small flash disk that replaces the hard disk (16M) or the small screen (800x600). Whatever the reason, I never really got these machines doing much more than booting into a command prompt. They’re still sitting in my basement gathering dust. One day I’ll revisit this project!

My second attempt was to just use a laptop. Unfortunately Linux (at the time) didn’t do very well for sleep modes and these older machines (the newest laptop I have is a Celeron 2.4Ghz Dell Inspiron 1100) take a fair time to boot. Windows… well, see the above comments! Also didn’t help that this machine only had 256Mb of RAM, so Windows was pig slow (even now I’ve upgraded it to 1Gb of RAM it’s still slow! Very slow hard disk, it seems).

My third attempt was to use a netbook. I originally got an Eee700. This suffers from a small 7” screen, small keyboard etc but in most respects it could be hacked into doing what I wanted… worst case I’d put on a different Linux distribution (Ubuntu netbook?). The problem was actually that this machine better filled the niche of “guest PC” for the living room; if someone wanted to check on email or surf the net or look up IMDB during the Oscars or whatever then this was a perfect machine! It was too useful to be relegated to library use :-) In fact I ended buying a 2nd (refurbished) Eee (an Eee900 this time) which I have at work connected to the guest wireless network so I can access personal email etc from work. I like these EeePCs!

So now attempt 4. I managed to get hold of a HP t5720. These actually come in two variations; the t5720 which runs Microsoft Windows XPembedded, and the t5725 which runs a Linux variant (based on Debian Etch). The hardware seems identical, it’s just the OS that they boot into that distinguishes them. These can be obtained for around $100 from eBay. Now these machines were designed as thin clients.

Inside they have a 1Ghz AMD Geode 1500 processor with 256Mb RAM and 512Mb of flash disk (instead of a hard disk). The memory can be upgraded using SODIMM memory. There’s a plethora of USB ports, built in audio, PS/2 keyboard/mouse ports… in other words it looks pretty much like a bog standard (underpowered) PC. Interesting there’s no moving parts at all; it’s all passively cooled with a couple of massive heat sinks on each end of the case.

I plugged in a USB-DVD drive and booted (“rescue mode”) from a CentOS DVD I had around and confirmed this could read the hardware normally. Given the small amount of memory and disk I don’t think I’ll install CentOS on this machine,but this is powerful enough that I should be able to run a real OS without much problem. I just need a quick boot time!

Looking around at the different Linux thin-client builds it seems there’s quite a few options to chose from. Firstly we have the standard HP build, which can be downloaded from their website. This is based on Debian Etch. It’s pretty old, now. Installation was simple enough; make a bootable USB drive, let it boot, and it completely overwrites the internal flash drive with the new image. It’s slow, but it works. Remove the USB device, reboot… and we have a Linux install. From POST completion to the OS being ready for use takes around 30 seconds.

By default the machine drops straight into a local X configuration; from here you can run firefox (sorry, iceweasel) - but it’s an old version. Now, it looks like HP have added some of their own APT repositories, and also left references to a main etch respository so I thought I’d try to update the machine. Here the limits of the small hard disk show up. “apt update” worked just fine. “apt upgrade” complained about the Debian official packages not being properly signed (I guess HP didn’t put the right public key in the image) but otherwise worked… until the disk filled up. Attempting to recover from this ended up with an unbootable system. Umm. Perhaps someone more versed in Debian recovery than me could have got it working again, but I didn’t need to; I just reflashed it again from the HP image. This time round I just updated iceweasel. However this still only took me to 2.0.0.19, which is very old in firefox terms. Hmm.

The HP image can also work as a more general “X terminal”, where you connect to a remote XDMCP server, authenticate and then run the X session on the remote machine. The desktop macune just runs the X server. In this mode it doesn’t really matter than the provided software is old because you don’t really run it. I’ve not tested this option yet; I need to work out how to configure the display manager on my server to accept requests from remote machines but not run X on my local console (‘cos I prefer a text terminal on my server consoles, thank you very much). I think that if I’m going to use the HP image then I’ll have to investigate this mode more. I’ll also have to determine how well audio works, and how quickly (slowly?) the system starts up.

Another way of doing terminal services is the Linux Terminal Server Project. This is pretty much designed to control and manage diskless workstations. Unfortunately I started to get a headache just reading the site. It seems that the codebase is currently being redesigned (from LTSP4 to LTSP5) and the documentation isn’t always clear in what it’s describing. It also looks like it might be a lot of work to get it running on my existing CentOS server and needs network reconfiguration of the server. Umm. I’m not sure I want to go to all that hassle! So I’m currently giving LTSP a skip.

Previously I’ve used Damn Small Linux on tiny PCs (eg a laptop with 16Mb RAM). It’s amazing that it works so well, but it also looks like it’s stuck in the past (a 2.4 based kernel. I’d be surprised if modern software (firefox 3.5 for example) would run on it. There was a more modern version (DNS-N) but this also looks like there’s been no update for years.

Then there’s ThinStation. This looks interesting. Currently I’m testing this via PXEboot (network boot) so I can switch between this and the HP image just by rebooting. This took 50 seconds to boot but some of that delay may be due to needing to download the image. Of course booting from the network with this image gives a fresh start each time, so there’s no history. It runs a very basic setup using iceWM, has firefox 3.0.x and has many forms of connectivity (eg rdesktop, ssh, Citrix ICA, NoMachine NX, VMware view etc etc etc). I need to see what power-saving functionality is available on this build, or whether I need to shutdown/reboot each time, and whether the local printer/scanner functionality will work.

Next up is xPUD. This is almost a web-based OS. It comes as a kernel image with a 44Mb initrd image attached (so it takes a little while to netboot because there’s a bit of data to transfer!) but once it’s got the data into RAM it boots fast; under 10 seconds from kernel startup to running system. It’s a different look’n’feel, but it seems to work. The main downside is that the ethernet controller on this HP (a via-rhine) doesn’t have a driver in the base OS image so if I netboot then it won’t find the ethernet controller so I can’t actually see the internet! If I boot from a USB flash drive then I can put the optional drivers package on the drive as well, and this has the via-rhine driver needed. Firefox is 3.5.5, which is prety new. Attempts to update to the latest, though, don’t appear to work. And is probably pointless given that a reboot would just rollback to the downloaded image. This is something I need to be aware of with xPUD; if I want newer application versions I may end up having to roll my own. But it’s interesting and something to look at.

And this is where I am at the moment; playing with the HP/Debain build, ThinStation and xPUD.

Of course I don’t have ethernet in the library, so I need to go wireless. This adds a new dimension… some USB wireless cards are only supported under newer kernels and these images (eg the HP one) may not have the required drivers. So do I go with a wireless USB card, or do I get a WRT54GL (or compatible) running a firmware such as Tomato in “access point” mode so it acts like a bridge, effectively providing ethernet to that room?

At the moment I’ve got my laptop in there running Windows (sigh) I think I’ve got a way to go on this project, yet!