I have read a lot of documentation. Some documentation is great, some is less than great, and some is actively hindering you by feeding you false information. There is little point in talking about good documentation, as it will just do what it should and lead you gently towards your desired outcome, but there are generally three types. They can be summed up as “good”, “bad”, and “actively misleading”.

The good one you can recognize by how it has a gentle introduction, one or more well-presented examples of use, and when you look for the more advanced parts, they are where you expect them to be. If you are a big fat nerd like me, you probably have some favourite documentation of your own. But talking about good documentation is not that exciting.

Bad documentation, on the other hand, is frustrating and fills me with great anger, and is more interesting to talk about. There are two main categories of bad documentation, each with their own subcategories.

The first is when there is documentation, and it is correct, but it is extremely minimalist and leaves a lot open to interpretation. A good example of this is perhaps a piece of software where all the most necessary parts are laid out, but there are no examples, no introduction to gently get you started. This leads the would-be user to go to Reddit, Stack Overflow, or now, to an LLM (better hope whatever you want to do is pretty simple and common) for help with implementation.

The second is when someone wanted to write a very wordy book, but was only allowed to write technical documentation, and they decided to take that out on the sorry reader of said documentation. Usually, this also involves going deep into subtopics which have their own pages with links to other pages, and so on. Good examples of this are Oracle and Microsoft documentation. The documentation is correct, but more confusing than it should have been. When reading for Azure exams, I often go to other places where the same information is presented much more succinctly and understandably. I can then go back and read the official documentation for specific details.

For the product owners, this is also a symptom of a disease and means you should burn it all down and start over. Thank you. The bad documentation is, however, mostly annoying; it does not hinder the eager sysadmin or programmer, it only slows us down momentarily.

The actively misleading documentation is where I start to get thoughts about taking the keyboard away from the author and smacking them with it until they start using it right.

A very recent example of this was while attempting to spin up a Docker container running some AI stuff in an air-gapped environment. This means you have no internet access, and so need to jump through a few hoops to get containers and the other data you need onto the machine you are working on. Standard procedure is to simply select the container you want to run in a nice little dashboard. It then fetches and runs the container for you. In the air-gapped environment, it cannot reach the Docker container repository, so you need to make your own little local version. Thankfully, this part is very easy. Just save the container on an internet-connected machine, transfer it to your secure server, and do “docker load” on the image file. So far so good, we can now run the local image by manually doing docker run, but we don’t want that. We want integration with the dashboard mentioned earlier. Thus, we follow the instructions that say to create a file called

$SPLUNK_HOME/etc/apps/mltk-container/local/images.conf

[my_custom_image]
repo = registry.local/
image = golden-cpu-custom:5.2.0
runtime = none
short_name = Golden CPU Custom

This lets Splunk DSDL know about our local images and is easy enough to do. After restarting Splunk, we go to our dashboard to select our custom image container aaaaaand the interface is grayed out and disabled… I usually have a lot of faith in the product owner and their documentation—too much, usually—because now comes the routine of going over the documentation again, going over the steps again, and making sure filenames, file contents, file permissions, etc. are all correct while growing agitated. You have followed the documentation, it should work, but what if the documentation is working against you? What if the documentation is WRONG? If you know something like Splunk well, and are very clever, you might think to check the structure of the dashboard page itself. I was not, and did not, but thankfully a colleague met the criteria and discovered that the page checked for the existence of the lines in the file and used it to populate the dashboard.

repo = registry.local/
image = golden-cpu-custom:5.2.0
runtime = none
short_name = Golden CPU Custom

The first three lines were fine, but “short_name” did not exist. It had been replaced by “title” and without that, the dashboard was frozen and unwilling to cooperate. This is something that is never an issue in the default solution and thus was likely not discovered. This type of issue is very easy to have. It is a small change in some lines for a dashboard that had unintended consequences and would require end-to-end testing to discover. Because of these reasons, I will not put too much negativity towards Splunk in this instance, but it was frustrating to follow a guide that was wrong, and would lead you to having a non-working setup.