In The Toolbox – Libraries, Console Apps & GUIs
Generally speaking software tools come in all shapes and sizes from one-line shell scripts [1] through to office “productivity” suites and beyond. If you’re someone who uses a computer to do a non-IT related job you’re more likely to sit at the far end of the spectrum plugging away on a GUI-based line-of-business (LOB) application interspersed with copious context switches to and from your email client, word processor and spreadsheets.
On the other hand if your primary focus is within IT itself, perhaps as a system administrator, tester or programmer you’re more likely to use a selection of tools that runs the full gamut. As I’ve illustrated in a past episode of this column about the “simple” task of finding some text [2] the nature of the tool you use also takes on a different form depending on the exact nature of the problem. Not only does the choice of tool depend on what outcome you’re looking for but it also may be driving how much you know about the problem domain itself. For example a GUI based XML document viewer might allow for more freedom of navigation whilst also hiding superfluous details so that you get to see the overall document shape, whereas a simple flat text editor or command window often generally gives you all the detail until you figure out how to reduce it.
Jeffrey Snover [3], one of the principle architects of PowerShell, once said the difference between GUIs and command line tools is about learning versus automation. For a non-programming oriented audience I’ve found that to be a very useful distinction, however for those people like myself who have a penchant for programming and home-grown tools I’d like to extend his definition slightly further:
As we shall see later the addition of libraries into the mix has a knock-on effect on how we think about designing systems from the perspective of supporting them later.
Like many programmers I found the graphics capabilities of computers intriguing and it inspired me to learn more about them. The ease with which you could draw a graph on an 8-bit home computer back in the 80’s rather than stare at a table of numbers was one of the reasons I got into this game. As that moved on to animating sprites and eventually 3D models it was no surprise that my first professional programming gig involved working on graphics applications – a drawing package, desktop publisher and associated graphics tools.
Although the dominant force for graphical applications, at least in the early days, was probably down to the promise of WYSIWYG for media creation, it also provides a way to represent more abstract concepts in a visually pleasing way that affords the user a fresh perspective on the task at hand. For example although folding text editors and syntax highlighting might add some extra pizazz to a mundane XML text document and allow the basic structure to shine through a little, a more dedicated tool that uses icons for the tree structure and separates out the element’s attributes might make comprehension easier. Throw in a find first / next feature that supports XPath expressions and you can explore the document in a different way that may gel where a manual navigation approach has meant you can’t see the wood for the trees.
Graphical diagnostic tools can also be less intimidating to a more casual user and may even stand-in where the primary product fails. I’ve had a GUI based integration test harness get shipped to beta testers because it provided a means of working around temporary failings in the core product. It’s also not the only time I’ve seen test harnesses take on a life outside the development team [4]. Any tool which is written to support development is almost certainly going to be useful to support the production system too, although you way want to add some extra features to try and minimise embarrassing mistakes [5].
Naturally we’re not just limited to classic thick client applications here either. Services can expose an endpoint for control and monitoring, perhaps through an HTTP endpoint on the side, and the meteoric rise of the browser means that client-side HTML and JavaScript has its uses. If the team has a particular preference for one brand of browser then plug-ins are another way of leveraging someone else’s product to put a face on your custom functionality.
GUIs may be great for interactive use but are generally a nightmare to automate. When you’re trying to automate tasks you really need tools specifically designed for automation. Even though a UI like TortoiseGit/SVN provides access to its feature via a command line process called TortoiseProc you just know that a message box popping up on an unattended server is going to ruin your day. Then you’re into the arms race of trying to download and run a tool which watches for (and automatically presses the “OK” button on) errant message boxes when they appear.
Even OLE Automation which, as its name suggested, promised to allow GUI based office tools like Word and Excel to be programmatically driven didn’t quite work as reliably unattended as was perhaps hoped for. In the finance world where Excel is the ad-hoc pricing tool of choice, many man-years of development time has been wasted trying to automate spreadsheet calculations (server-side) on the expectation that that task was easier (or cheaper) than getting a developer to code up the same logic up in a “real” programming language. These days the Excel “calculation engine” is more accessible but one wonders what the total cost of ownership has been for some of those earlier Heath Robinson approaches and whether it really paid off in the end.
The UNIX philosophy of small, simple command line tools, composed by passing textual output from one to another is still very much alive and well today. Driving these tools effectively is often for the more seasoned user as finding the right incantation of switches and syntax might only be obtainable with a solid read of the manual (or Stack Overflow) and plenty of examples. Although the industry is slowly converging on common patterns of switch indicators and naming conventions there are still far too many cases where case-sensitivity and the ability (or not) to merge multiple single character switches will trip you up. Go and PowerShell, with its single dash and long name convention, are notable exceptions to the typical dual switch approach (i.e. short and long form). Of course on Windows you’ve always had to play the game of “slash or dash” as you’re never quite sure if you’re dealing with a native Windows tool or a UNIX port. The rise of PowerShell and the new Windows Subsystem for Linux (WSL) will hopefully continue to educate Windows programmers on the correct choice of command line switch character.
The new-fangled Continuous Integration / Delivery / Deployment movement has definitely helped to promote a more automated software development environment and therefore stitching together sequences of tools either with pipelines or through scripting is more essential a skill than ever. Even though there are a number of products out there that aim to give you a drag’n’drop approach to creating and managing your build pipeline I still find a single build script that can run on both a developer’s workstation and CI server side to be preferable to two different build systems. In my experience the developer one is only a pale imitation of the real one and therefore awkward to reproduce and debug, and leads to “poisoning” of the build machine as more and more junk gets installed.
Despite the plethora of desktop products which provide you with a GUI for automating all manner of tools – GUI or command line based – which are no doubt a boon for the less tech savvy, a shell still seems to be the most popular choice for sequencing operations by the IT crowd. We shouldn’t be surprised as scripting is programming after all, just with much “heavier” functions. Ultimately unless something is only going to be run on the machine you developed it on you need to consider what capabilities the target host has and that will likely be the de-fact shells. Factor in trying to get the licensing sorted for any 3rd party product and the path of least resistance wins almost every time.
In the traditional taxonomy of software tools, which is what Jeffrey Snover was no doubt aiming at, libraries do not feature as they were never really usable out-of-the-box. Native code libraries packaged in their static library form are intended to be consumed at build time which means someone needs to write a host application for them. On the other hand dynamically linked libraries could be consumed directly if you knew what entry point to poke, what arguments to pass and how. On Windows this is possible to a degree with the RunDll32 utility but it’s rarely used in practice as little things like calling conventions make this a dangerous game to play. You could invoke a void function taking no arguments fairly safely but anything else was tantamount to running around with scissors.
The Component Object Model (COM), along with a few other technologies like Visual Basic and the Active Scripting Engine, allowed libraries to become scriptable “components” by standardising the calling convention and adding some metadata to make them a little more discoverable; not quite “--help” territory but a little better. The introduction of .Net and PowerShell took the idea a step further, although one shouldn’t discount the various other Windows offerings such as IronPython and F# as suitable scripting hosts; it’s just that they aren’t installed by default.
It might seem as though the humble static library has been short-changed. The problem is that to consume these you need to have a compiler and native programming skills. But that is exactly what customisation is all about – taking the core functionality for a tool, like Git, and sticking your own console application or UI on top of it. Or, as we’ve already mentioned above with the Tortoise family you might do both. Git is a prime example because its complexity makes it hard to grasp but the success of the TortoiseGit tool is almost certainly down to bringing familiarity to those used to the simpler concepts of TortoiseSvn.
For languages that provide interop with C, and let’s face it the most common ones do, you have the option to provide plug-in shims that allow the native functionality to be exposed through an adapter and consumed by the plethora of modern, non-native (e.g. JVM) based development tools like IDEs and CI services. Pure versions often arrive later but in cases like the Git TFS bridge where the back-end libraries are proprietary it may be the only choice. It must surely be a testament to the Git client architecture that adapters for so many different VCS products have been produced that give a reasonably seamless experience.
Passing credentials into Git is an excellent example of where the strength of a good library comes in. When dealing with private repositories the ability to securely drive Git in your build pipeline without exposing your credentials in log files and environment variables was originally quite a challenge as many of the supposedly mature, off-the-shelf choices struggled with this requirement. Throw in the new-fangled “submodules” feature and it started all over again as we needed to pass different credentials along the submodule pipeline. For a while I seem to remember switching between various Git plugins and command line tools in search for the right combination that worked with our team’s “Enterprise” set-up which threw in an NTLM-based proxy for extra friction.
Oranges are not the only fruit and these libraries (static and dynamic) aren’t the only two kinds either – anyone working in the Perl, Ruby, Python, Node.js, etc. space will quickly point out that they have had a solution for consuming libraries easily for years, although they tend to talk about “packages” rather than libraries. As we know [6] these can be fragile ecosystems at times if not treated with respect due to the ever growing volume of dependencies that get pulled in as we struggle to avoid reinventing the wheel.
For development purposes where I might need a solution to a short lived problem the reward can far outweigh the risks, but having been on the roller coaster journey of breakages and fixes in areas where you would prefer to have more stable products there is much solace to be gained from simply copying a single, small binary around. In a production support scenario I’m even less enamoured by the idea of downloading half the Internet just to run a tool. In a cloud hosted environment where you typically restrict both inbound and outbound network access this would be a non-starter.
Going back to the UNIX philosophy one wonders if the notion of “small” is only considered by some to apply to the amount of functionality it provides, not the size of the ecosystem required to create, distribute and execute it.
One of the oldest concepts we should be cognisant of when building software systems is that of modular design. Breaking a system down into small building blocks that separate out concerns to try and manage complexity is an on-going battle which programmers face every day. The introduction of lower-levels of testing, e.g. at unit and component level, have definitely had an effect in this area, but another driver you might want to consider for your design is the need for custom tooling.
Systems with plenty of moving parts – databases, queues, services, etc. – have many points of failure, or alternatively, points of diagnostic access. Although it may be built entirely on open protocols and products that doesn’t mean the lowest common denominator tools are the best tools for poking around, e.g. CURL for REST APIs. We build our system out of these building blocks because we have some value to add to them, hence our tooling should add value too. For example an HTTP based API that also needs authentication for its requests soon becomes tiresome as every invocation will require at least two requests – one for the token and one for the actual operation. Wrapping this up in a script is an easy win but if you’ve built a richer proxy for use elsewhere in your system then stick a simple CLI on top and you have the makings of a deployment smoke test and support tool.
What makes this type of tooling better is that you’re leveraging the same well-tested production code in your support toolkit as what you use in the actual system. This really starts to pay off when you need to perform operations that would be considerably more dangerous if you were to hack away at the underlying 3rd party products using their more generalised tools. For example a database is just an implementation detail of a system and whilst you can poke around with a SQL tool you run the risk of violating invariants of the system that would be enforced if you went through your system’s code. An RDBMS can provide a certain degree of protection from “tinkering” if you use its referential integrity features to the fullest whereas document databases and file-systems expect the application to do most of the heavy lifting to ensure the data remains semantically valid.
To give a more concrete example imagine having a service where you handle versioning of persisted data by upgrading it at read time. Over time you end up with objects of different versions and therefore lots of code and tests to handle the various incarnations. It can get trickier to make changes as you start to consider all the variants, plus looking directly in the data store during support gets harder as you have to factor in all possible versions into your queries. If you could upgrade all the objects to a new baseline version you could then remove a lot of complexity, i.e. dead code and tests. The production code already exists to do the job; you just need to be able to access it. Although you could add a new private service endpoint you should be able to just host the underlying logic in a simple process container (e.g. scripting host) that can slowly walk the object store and call Load() followed by Save() for the instances of object that need uplifting.
By building the logic into the production code rather than the support tooling you naturally reduce the risk of a tool getting out of step with the real code and then corrupting something accidentally later. (How many people have tests for their tools?)
Aside from their primary purpose – to accomplish certain tasks more easily – custom tools also provide a useful auxiliary function which is that their lower-level focus can help with learning a new codebase. Learning a complex system comprising of many moving parts can be a daunting experience as you can struggle to find a place to start looking. Whilst the natural place might be the UI as the user is ultimately the primary point of focus, with a reporting engine however working backwards towards the inputs can be hard work. The tools won’t really help you get an overview of a system but they can be a good way to help understand how part of a system works in isolation. (Obviously a suite of component level tests would provide an even better source of discovery but rarely do you find many tests in between the unit and end-to-end levels.)
In this scenario I’ve found command line tools much better for this purpose than GUI based equivalents. What makes tests and command line tools preferable is that they both generally allow you to take a “straight line” through a piece of functionality. In essence they gather the inputs, invoke the behaviour, and do something with the outputs – exactly the same kind of approach that makes reasoning about pure functions easier. In contrast GUIs are inherently event based and therefore much of the functionality used to prepare the inputs are scattered about in different event handlers with state built up piecemeal before the final push. If the code is not well factored, and we’re talking about a custom tool here so attention to detail is less likely, then it can be harder to separate the GUI glue from the production code. For example you may have to keep one eye on the call stack or look at what namespace bits of code belong to to know if you’re inside or outside the production code for the component of interest.
In my early days as a professional programmer my preference would have been to opt for a GUI based approach, which is probably understandable as what I was working on were GUI based products. Many of the early “services” I worked on also had a built-in UI to act as a sort of “administration console” as it was far easier than building in some form of IPC and separate UI tooling back then. It wasn’t until a little while later when I started working on more traditional heavily distributed “headless” services that I started to truly appreciate the power of well factored libraries, command line tools and automation. However I still built a couple of UI based test harnesses because manual testing was still the dominate approach.
Despite a move into REST APIs for a period, which is clearly a server-side role, I’ve still had to keep my eye in on the UI based tooling as showcasing the API features both to the product owner and wider shareholders is much easier with a more user friendly approach [7]. While using classic REST tools and Swagger work up to a point, the moment the “user” journeys involve any kind of authentication the implementation details start to dominate the proceedings. A “demo UI” can also act as an exploratory testing tool for the API too so it has value outside of showcasing new features.
What is becoming more apparent to me however is that it is all too easy to stick to what you already know and create a tool which serves your immediate needs without considering its audience and whether more than one user interface might be desirable because the same the same tasks straddle the lines of analysis, development, testing, deployment and support. It’s highly understandable of course because it’s probably just a yak you’re shaving and a means to a different end than the feature you’re supposed to be delivering to your customer. Don’t forget though that non-functional user stories have considerable value too and that maybe it’s just a matter of prioritisation.
Gerry Weinberg [8] suggests that you don’t really understand a tool until you know how to abuse it at least 3 different ways. I don’t know if trying to automate Excel on a server counts as one of your three but it definitely qualifies as abuse. One presumes he was talking about the more functional aspects rather than the friction generated by having the wrong form of interface. Either way it’s important that we pay attention to what toolkit we’ve built for our system as much as the end product we’re using it on. And, if we get the design of the system right, the addition of custom tools shouldn’t be much of a burden on delivery as the functionality we need to expose will only be a few lines of code away. Our final consideration is then how we’re looking to interact with it and whether automation may be desirable, along with learning, exploration, or all those things, but in different ways.
[1] In The Toolbox – Wrapper Scripts, Chris Oldwood, C Vu
25-3,
http://www.chrisoldwood.com/articles/in-the-toolbox-wrapper-scripts.html
[2] In The Toolbox – Finding Text, Chris Oldwood, C Vu 27-6,
http://www.chrisoldwood.com/articles/in-the-toolbox-finding-text.html
[3] Wikipedia, https://en.wikipedia.org/wiki/Jeffrey_Snover
[4] From Test Harness to Support Tool, Chris Oldwood, blog,
http://chrisoldwood.blogspot.co.uk/2012/11/from-test-harness-to-support-tool.html
[5] Afterwood – Honesty, Chris Oldwood, Overload 139,
http://www.chrisoldwood.com/articles/afterwood-honesty.html
[6] Afterwood – Knocked For Six, Chris Oldwood, Overload
136,
http://www.chrisoldwood.com/articles/afterwood-honesty.html
[7] Showcasing APIs – The Demo Site, Chris Oldwood, blog,
http://chrisoldwood.blogspot.co.uk/2016/06/showcasing-apis-demo-site.html
[8] An Introduction to General Systems Thinking, Gerald Weinberg, 1975 / 2001, ISBN 0932633498
Chris Oldwood
21 April 2018
Chris is a freelance programmer who started out as a bedroom coder in the 80’s writing assembler on 8-bit micros. These days it's enterprise grade technology in plush corporate offices. He also commentates on the Godmanchester duck race and can be easily distracted via gort@cix.co.uk or @chrisoldwood.