Mike Slinn
Mike Slinn

dotenv For Command Line Interpreters

Published 2020-10-24.

This article is categorized under Bash

I’ve built a shell that allows me to be extremely productive while writing technical documentation. The generated documentation displays code examples quickly and easily. This article discusses the shell, dotenv-python.

JSON Output

Many command line programs return JSON. Examples include:

  1. ansible.netcommon.cli_command
  2. Amazon Web Services CLI
  3. Cloudflare Workers CLI
  4. Databricks CLI
  5. Digital Ocean CLI
  6. Docker CLI
  7. Google Cloud CLI
  8. Kubernetes kubectl CLI
  9. Linode CLI
  10. Microsoft Azure CLI
  11. Mulesoft Anypoint Platform CLI
  12. Rackspace CLI
  13. Salesforce CLI
  14. Scaleway CLI
  15. Twilio CLI
  16. Zoom Rooms

I work with JSON for two reasons:

  1. To interact with a server, such as those listed above. This is repetitive, tedious and boring.
  2. To write documentation about how to work with servers. This is repetitive, tedious and boring.

Easy to Manage, Not Boring, Tedious or Repetitve

It is difficult to keep track of CLI output and manage it over time. To keep track of returned name/value pairs, and to persist them, I wrote a Python library called mem. This article shows how mem uses hidden files called .env in concert with environment variables to store and retrieve names and values. Files called .env are used in Twelve-factor webapps.

Twelve-Factor Apps and dotenv-python

Storing configuration in the environment is one of the tenets of twelve-factor web applications. Anything that is likely to change between deployment environments, such as resource handles for databases or credentials for external services, should be extracted from the code into environment variables. However, it is not always practical to set environment variables on development machines or continuous integration servers where multiple projects are run. Instead, dotenv-python loads variables from an .env file into the environment when the application starts.

The GitHub Repository for mem

The code lives in a GitHub repository.

Installation

TODO write me.

About mem

mem saves key/value pairs in a hidden file in the current directory called .env. This is useful when working with commands that generate JSON. Each of your projects can have a different .env file, which means that the key/value pairs defined within them are specific to the projects that they are located with.

mem works two ways:

  1. By receiving the name of a new environment variable and its value, specified separately.
  2. By receiving the name of an existing environment variable that has been exported.

Once a key/value pair is defined in .env, it becomes available after reading the file back in using the source command. The alias defined above reads the file back in automatically. The alias display the new or modified name/value pair that it just defined.

System Clipboard Support

mem also works with the system clipboard. This greatly speeds up the process of writing documentation generated by Jekyll, which is used to create this website.

  1. It copies a block of HTML code to the system clipboard each time it runs. The HTML invokes Jekyll plugins I wrote called pre and noselect. For example, if you run mem like this:
    mem NAME VALUE
    Then after the mem script completes the clipboard might contain something like this:
    {% pre copyButton %}{% noselect %}mem NAME "$(
      COMPUTATION
    )"
    {% noselect NAME=VALUE%}
    {% endpre %}
    
    Which renders as:
    $ mem NAME "$(
      COMPUTATION
    )"
    NAME=VALUE
  2. If the system clipboard is not empty when mem runs then the COMPUTATION placeholder shown above will be set to the clipboard contents. Otherwise the value of the COMPUTATION placeholder will be fill_in_the_computation.

Usage

Here is the help message for mem:

$ $mem/src/main.py -h 
 usage: main.py [-h] [-a DIALOG_FILE_NAME] [-b] [-c] [-d] [-e DOTENV_FILE_NAME]
               [-f {jekyll,html5,markdown,plain}] [-H] [-p] [-u {qt,cli}] [-v]
               [variable_name] [command_or_value]

Persists environment variables (names and values) to a file. By default the
file is called .env and is found in the current directory. The original
command can also be captured when run as a shell. Environment variable
substitution and subshells can be used for constructing values.

mem copies a block of markup code to the system clipboard each time it runs,
to speed up the process of writing documentation. Supported formats are
Jekyll, Markdown, HTML5 and plain text.

mem can either present a QT-based gui or a command-line interface.
The CLI mode provides two sub-modes:

1) A name and value are provided on the command line that invokes mem.
   SYNTAX: mem [OPTIONS] environment_variable_name command_or_value

   This mode is automatically entered when environment_variable_name and
command_or_value
   are both provided. The -u/--ui switch is ignored.

   EXAMPLE:
   $ mem DIR pwd
   DIR=/mnt/_/work/mem
   # The value of CPU is available to subsequent mem invocations as "$DIR"
   $ mem FILES ls $DIR
   FILES='bin/  data/  docs/  LICENSE  __pycache__/  README.md  src/  tests/'

2) No parameters are provided on the command line that invokes mem.
   In this mode mem loops continually, prompting for name / value pairs.

   SYNTAX: mem [OPTIONS]

   EXAMPLE:
   $ mem
   mem: name> DIR
   mem: new value for DIR> /etc
   DIR='/etc'
   # The value of DIR is available to subsequent mem invocations as "$DIR"
   $ source .env
   $ echo "$DIR"
   /etc
   mem: name>  # Press Enter, ^D or ^C to exit.

For more information about mem, see
https://mslinn.com/blog/2020/10/24/working-with-command-line-interps.html

positional arguments:
  variable_name         Optional name of environment variable
  command_or_value      Optional command or value of environment variable

optional arguments:
  -H, --heading         Prompt for heading.
  -a, --append DIALOG_FILE_NAME
                        Append the cli dialog history to specified file
  -b, --clipboard       Copy formatted example code to clipboard after every
                        definition.
  -c, --comment         Prompt for comment (paragraphs).
  -d, --debug           Enable debug output
  -e, --dotenv DOTENV_FILE_NAME
                        Dotenv file to read/write keys & values.
  -f, --format {jekyll,html5,markdown,plain}
                        Format to save dialog in. Markdown uses GitHub syntax.
                        Jekyll format requires Mike Slinn's plugins.
  -h, --help            show this help message and exit
  -p, --append_previous
                        Continue appending to the most recent markup file.
  -u, --ui {qt,cli}     User interface style.
  -v, --version         show program's version number and exit

Copyright 2020 Michael Slinn. All rights reserved.

A very common usage pattern is to provide a computed value computed within a subshell, like this "$()". For example, define a new environment variable called TODAY, with the value computed by "$( date )":

$ mem TODAY "$( date )"
TODAY='Tue Oct 27 10:18:18 EDT 2020'

Loading Name/Value Pairs

To restore the names and values in this shell, or another shell, make the directory current containing the .env files that define the name/value pairs for a project. Type:

$ source .env

This means you can walk away and resume the session anywhere the .env file is available without losing context.

Working with JSON

I frequently use jq for parsing JSON in the bash shell. Install it like this:

$ sudo apt install jq

Single Quote Literal Strings

This example saves some JSON in .env and extracts a value from it later. JSON surrounds names and string values with double quotes. In the following example single quotes surround the value, which is specified as a literal string. This is so the JSON literal string will be stored as a single string in .env. If double quotes had been used to enclose the literal string of JSON then the double quotes inside the JSON literal string would have needed to be escaped.

The <<< in the following command history pipes the string defined to the right of the <<< into the jq process as stdin.

$ mem JSON '{"name1": "value1", "name2": "value2"}'
JSON='{"name1": "value1", "name2": "value2"}'

$ jq -r .name1 <<< $JSON
value1

$ jq -r .name2 <<< $JSON
value2

Double-Quote Subshell Output

Bash subshells are useful for computing values. They are defined by writing a dollar sign followed by parentheses, like this: $() If there is a possibility that the value might have whitespace in it, or special characters, the subshell should be enclosed in double quotes, like this: "$()".

The double quotes in the following code example ensure that the JSON returned by curl is stored as a single string in .env.

$ mem JOKE "$( curl -s http://api.icndb.com/jokes/random )"
JOKE='{ "type": "success", "value": { "id": 327, "joke": "Chuck Norris once ordered a steak in a restaurant. The steak did what it was told.", "categories": [] } }'

$ jq -r '.value.joke' <<< $JOKE
Chuck Norris once ordered a steak in a restaurant. The steak did what it was told.

You could also use backticks (``) instead of "$()":

$ mem JOKE `curl -s http://api.icndb.com/jokes/random`
JOKE='{ "type": "success", "value": { "id": 327, "joke": "Chuck Norris once ordered a steak in a restaurant. The steak did what it was told.", "categories": [] } }'