Production Infrastructure Scripts

Published 2020-03-01. Last modified 2024-12-15.
Time to read: 2 minutes.

This page is part of the av_studio collection.

The Production Infrastructure Overview and Production Infrastructure Directories articles describe the environment that the scripts on this page operate within.

All of these scripts require Ruby to be set up; please see Setting Up a Ruby Development Environment.

Create Song Structure

The following script creates the subdirectory structure for a song. Any pre-existing files in the directory are moved to their appropriate subdirectories. A private GitHub repository is created for the song.

Use the A Streamlined Git Commit to ensure that only files that fit in git are checked in.

Installation

Install dependencies:

Shell

$ gem install colorator

$ gem install mime-types

Save the code as song_create in a directory on the PATH, such as /usr/local/bin/.

Make song_create executable:

Shell

$ sudo chmod a+x /usr/local/bin/song_create

Example Usage

This script can be called from any directory.

Shell

$ song_create $songs/test
Creating a new song subdirectory structure in '/mnt/e/media/songs/test'
  /mnt/e/media/songs/test/daw
  /mnt/e/media/songs/test/dvrArchives
  /mnt/e/media/songs/test/inputs/image
  /mnt/e/media/songs/test/inputs/video
  /mnt/e/media/songs/test/outputs/audio
  /mnt/e/media/songs/test/outputs/stems
  /mnt/e/media/songs/test/outputs/youTube
Opening E:\media\songs\test

Source Code

song_create

#!/usr/bin/env ruby

require 'colorator'
require 'fileutils'
require 'mime/types/full'

# https://github.com/mime-types/ruby-mime-types/
# gem install 'mime-types'

# Creates a new music video (song) subdirectory structure.
# Does not create any files, so this script is idempotent;
# you can accidently call it several times without problems.
#
# Creates this structure:
# ├── daw
# │   └── beats  # Experiments
# ├── dvrArchives  # DaVinci Resolve project archives
# ├── inputs  # For video editor (DaVinci Resolve) project
# │   ├── images  # QR code, etc.
# │   └── videos  # A and B rolls
# └── outputs  # Renders, mostly by DAWs and video editor
#     ├── mp3s  # Pro Tools renders
#     ├── stems  # Zipped audio files
#     └── youTube  # DaVinci Resolve renders
#
# Sample output:
# $ song_create $songs/test
# The subdirectory structure '$songs/test' already exists. No problem!
#   /mnt/e/media/songs/test/daw
#   /mnt/e/media/songs/test/dvrArchives
#   /mnt/e/media/songs/test/inputs/image
#   /mnt/e/media/songs/test/inputs/video
#   /mnt/e/media/songs/test/outputs/audio
#   /mnt/e/media/songs/test/outputs/stems
#   /mnt/e/media/songs/test/outputs/youTube
# Opening E:\media\songs\test
#
class SongStructure
  attr_reader :this_song_dir, :song_name

  def initialize(song_name)
    unless song_name
      puts "Error: No song name was provided. Aborting.".red
      exit 1
    end
    @song_name = song_name

    @songs_dir = ENV.fetch('songs') # root of all songs
    unless @songs_dir
      puts "Error: Environment variable 'songs' is not defined. Aborting.".red
      exit 2
    end
    if @songs_dir.empty?
      puts "Error: Environment variable 'songs' has no value. Aborting.".red
      exit 3
    end

    unless Dir.exist? @songs_dir
      puts "Error: Directory #{@songs_dir} does not exist. Aborting.".red
      exit 4
    end

    @this_song_dir = "#{@songs_dir}/#{@song_name}" # root of the new song's directory
  end

  def create
    if File.exist? @this_song_dir
      puts "The subdirectory structure '#{@this_song_dir}' already exists. No problem!".yellow
    else
      puts "Creating a new song subdirectory structure in '#{@this_song_dir}'".yellow
    end

    [
      'daw',
      'dvrArchives',
      'inputs/audio', 'inputs/image', 'inputs/video',
      'outputs/audio', 'outputs/stems', 'outputs/youTube',
    ].each do |dir|
      puts "  #{@this_song_dir}/#{dir}".yellow
      FileUtils.mkdir_p "#{@this_song_dir}/#{dir}"
      FileUtils.touch "#{@this_song_dir}/#{dir}/.gitkeep"
    end
  end

  def display_this_song_dir
    win_song_dir = `wslpath -w '#{@this_song_dir}'`.chomp
    puts "Opening #{win_song_dir}".yellow
    `cmd.exe /c 'explorer.exe /separate, #{win_song_dir}'`
  end

  # Creates a git repo with the same name as the song, but with the 'song_' prefix added if required
  def git_repo(this_song_dir, song_name)
    Dir.chdir(this_song_dir) do
      if Dir.exist? '.git'
        puts "The song in '#{this_song_dir}' already has a git repository".yellow
        return
      else
        `git init`
        name = song_name.start_with?('song') ? song_name : "song_#{song_name}"
        `gh repo create --private --source=. --remote=origin #{name}`
      end

      write_git_ignore
      write_readme

      `commit` # See https://mslinn.com/git/1050-commit.html
    end
  end

  # Moves files into position based on:
  #   1) Name
  #   2) extension if no mimetype
  #   3) mimetype (set by examining extension). The Mimetype gem only looks at the filetype,
  #      not the contents, so the file need not exist.
  def move_files(song_dir)
    Dir.chdir(song_dir) do
      Dir.glob('*').each do |entry|
        next if File.directory? entry

        filetype = File.extname(entry)[1..]
        mimetype = MIME::Types.type_for entry
        media_types    = mimetype.map(&:media_type).uniq
        media_subtypes = mimetype.map(&:sub_type).uniq

        subdirectory = if /README.*/.match?(entry) then next # Ignore this file
                       elsif /.*zip$/.match?(entry) then 'outputs/stems'
                       elsif media_types.include?('audio') && filetype != 'mp4'
                         if media_subtypes.include?('midi')
                           'midi' # Both an input and an output, so make it a peer of input/ and output/
                         elsif filetype == 'mp3' # This is probably an audio rendering
                           'outputs/audio'
                         elsif filetype.start_with? 'aif' # Might be a sample or a render
                           'outputs'
                         else # Sample assumed
                           'inputs/audio'
                         end
                       elsif media_types.include?('image') then 'inputs/image'
                       elsif media_types.include?('video') then 'inputs/video'
                       else # rubocop:disable Lint/DuplicateBranch
                         next # Leave unrecognized files where you find them
                       end
        path = "#{@this_song_dir}/#{subdirectory}"
        FileUtils.mkdir_p path
        FileUtils.mv entry, "#{path}/#{entry}", verbose: true
      end
    end
  end

  # This method is not required any more, but I keeping it just in case
  def rename_repo
    remote_origin_url = `git config --get remote.origin.url`
    repo_name = `basename -s .git #{remote_origin_url}`
    return if repo_name.start_with?('song')

    repo_names = `gh repo list --json name --jq .[].name`.chomp

    renamed_repo = "song_#{@song_name}"
    if repo_names.include? renamed_repo
      puts "#{renamed_repo} already exists, not modified.".yellow
      return
    end
    puts "Renaming git repo '#{@song_name}' to '#{renamed_repo}'.".yellow
    `gh repo rename #{renamed_repo} --yes`
  end

  # Assumes the file should be written to the current directory
  def write_file(name, contents)
    if File.exist? name
      puts "'#{@this_song_dir}/#{name}' already exists, so it was preserved.".yellow
    else
      puts "Writing '#{@this_song_dir}/#{name}'".yellow
      File.open(name, 'w') do |f|
        f << contents
      end
    end
  end

  # Assumes the file should be written to the current directory
  def write_git_ignore
    git_ignore = <<~HEREDOC
      *.Identifier
      *.tmp
      *~
      ~*
      .DS_Store
      desktop.ini
    HEREDOC
    write_file '.gitignore', git_ignore
  end

  # Assumes the file should be written to the current directory
  def write_readme
    write_file 'README.md', <<~HEREDOC
      # #{@song_name}

      Created #{Time.now.strftime "%Y-%m-%d"}
    HEREDOC
  end
end

if __FILE__ == $PROGRAM_NAME
  structure = SongStructure.new ARGV[0]
  structure.create
  structure.move_files structure.this_song_dir
  structure.git_repo structure.this_song_dir, structure.song_name
  structure.display_this_song_dir
end

Create Song Webpage

The following script creates a new web page on this website for a new song. Referring to the production overview image, it creates a new web page with the given name in the right-hand area pertaining to Jekyll.

A QR code is also created for the webpage URL, which begins with https://mslinn.com/songs/.

Installation

Install dependencies:
Shell
```
$ gem install colorator
```
Save the code as song_page_create in a directory on the PATH, such as /usr/local/bin/.

Make song_page_create executable:

Shell

$ sudo chmod a+x /usr/local/bin/song_page_create

Example Usage

This script can be called from any directory. It overwrites any previous .html file of the same name in the collections/_songs/ subdirectory of the website. Users should commit their changes before calling this script, just to be sure.

Shell

$ song_page_create my_song
Wrote /var/sitesUbuntu/www.mslinn.com/collections/_songs/my_song.html

Source Code

song_page_create

#!/usr/bin/env ruby

require 'colorator'
require 'fileutils'

# Creates $msp/collections/_songs/#{song_name}.html from song_template.html in the same directory.
# Overwrites the file if present.
# Creates a QR code image.
#
# TODO: update template contents with qr code and other strings. Right now manual editing is necessary
#
# Sample output:
# $ song_page_create space_truckin
#   Creating /var/sitesUbuntu/www.mslinn.com/space_truckin.html
#   Opening the new web page in VSCode
#
class SongPageCreate
  def initialize(song_name)
    unless song_name
      puts "Error: No song name was provided. Aborting.".red
      exit 1
    end
    @song_name = song_name

    @msp = ENV.fetch('msp') # root of mslinn.com (website)
    unless @msp
      puts "Error: Environment variable 'msp' is not defined. Aborting.".red
      exit 2
    end
    if @msp.empty?
      puts "Error: Environment variable 'songs' has no value. Aborting.".red
      exit 3
    end

    unless Dir.exist? @msp
      puts "Error: Directory #{@msp} does not exist. Aborting.".red
      exit 4
    end

    @webpage_dir = "#{@msp}/collections/_songs" # path of the directory containing the web page for the new song
    @template_path = "#{@webpage_dir}/song_template.html"
    unless File.exist? @template_path
      puts "Error: '#{@template_path}' does not exist. Aborting."
      exit 1
    end

    @webpage_name = "#{@webpage_dir}/#{@song_name}.html" # path of the web page for the new song
    @qr_image = "#{@webpage_dir}/images/qr_#{@song_name}.png"
    @webpage_url = "https://www.mslinn.com/songs/#{@song_name}.html"
  end

  def create
    if File.exist? @webpage_name
      puts "The webpage '#{@webpage_name}' already exists. IT WILL BE NUKED AND REPLACED NOW.".yellow
    else
      puts "Creating a new song webpage in '#{@webpage_name}' from 'template.html' in the same directory.".yellow
    end
    FileUtils.cp @template_path, @webpage_name
  end

  def display_dir
    webpage_name_win = `wslpath -w '#{@webpage_name}'`.chomp
    puts "Opening the directory containing #{webpage_name_win}".yellow
    `cmd.exe /c 'explorer.exe /separate, #{webpage_name_win}'`
  end

  # Overwrites
  def qr_code
    # yes | sudo apt install qrencode
    `qrencode -o #{@qr_image} '#{@webpage_url}'`
  end
end

if __FILE__ == $PROGRAM_NAME
  spc = SongPageCreate.new(ARGV[0])
  spc.create
  spc.display_dir
  spc.qr_code
end

Song Template

View the rendered song template for new songs, and examine its source code:

song_template.html

---
bpm: 120
history: 2024-11-28
date: 2024-11-28
key: C
description: "___, an original composition by Mike Slinn. All rights reserved."
guitar_pro_file:
headerImage:
last_modified_at: 2024-11-28
layout: songs
live_file:
mp3:
musescore:
musicxml:
pdf:
png:
pro_tools:
published: false
qr_code: /songs/images/qr_song_template.png
stems:
title: Song Template
---
<!-- #region about -->
{% capture about %}
<p>
  This is the song template.
</p>
{% endcapture %}
<!-- endregion -->


<!-- #region audio -->
{%- comment -%}
  {% capture audio %}
    <p id="mp3_long">
    </p>
    {% include audio.html mp3="/songs/mp3s/space_truckin.mp3" %}
  {% endcapture %}
{%- endcomment -%}
<!-- endregion -->


<!-- #region music_videos -->
{% comment %}
  {% capture music_videos %}
    {% include video.html youtube_id='6wMf1AHCJXE' %}
  {% endcapture %}
{% endcomment %}
<!-- endregion -->


{% include song.html
  about=about
  audio=audio
  change_log=change_log
  competition=competition
  how_to_play=how_to_play
  learning_videos=learning_videos
  lyrics=lyrics
  music_videos=music_videos
  planned=planned
  tools_used=tools_used
%}

Zip Stems

The following script packages audio stems into a zip file. It performs the functionality denoted by the dashed line with the label zip in the production overview image.

Pro Tools generates audio stems with a common prefix. The zip file created by this script is automatically named from the common prefix of all the audio stem files.

Installation

Install dependencies:

Shell

$ gem install colorator

$ gem install rubyzip

Save the code as song_stems in a directory on the PATH, such as /usr/local/bin/.

Make song_stems executable:

Shell

$ sudo chmod a+x /usr/local/bin/song_stems

Example Usage

This script can be called from any directory.

Shell

$ song_stems $songs/my_song
Wrote /mnt/e/media/songs/my_song/outputs/stems/my_song_stems_120_bpm.zip

The following is identical to the above:

Shell

$ song_stems 'E:\media\songs\one_year_older\outputs\stems'
Wrote /mnt/e/media/songs/my_song/outputs/stems/my_song_stems_120_bpm.zip

Source Code

song_stems

#!/usr/bin/env ruby

require 'colorator'
require 'find'
require 'zip'

# gem install rubyzip
# gem 'rubyzip'

# Zips audio files in songs\your_song\outputs\stems into
# $msp/collections/_songs/stems/ and deletes the original files
#
# If this program is run from either of these song directories:
#   /mnt/e/media/songs/one_year_older/
#   /mnt/e/media/songs/one_year_older/outputs/stems/
# then the defaults will overwrite any zip file in
# the corresponding www.mslinn.com directory
#   U:\var\sitesUbuntu\www.mslinn.com\collections\_songs\stems
#
# In other words, this invocation:
#   $ cd $media/songs/one_year_older
#   $ song_stems
# are the same as:
#   $ song_stems E:/media/songs/one_year_older
#   $ song_stems E:/media/songs/one_year_older U:/var/sitesUbuntu/www.mslinn.com/collections/_songs/stems
#   $ song_stems $media/songs/one_year_older
#   $ song_stems $media/songs/one_year_older $msp/collections/_songs/stems
#
# @param src [String] Can be either a Windows path or a WSL path
# @param dest [String] Defaults to common stems directory for all songs in mslinn.com
# Will be overwritten if it already exists.
class SongStems
  def initialize(src = nil, dest = nil)
    @src = src.nil? ? Dir.pwd : expand_env(src)
    unless File.exist? @src # Did the user provide a WSL path?
      # TODO: wslpath fails if @src does not exist; need to handle this
      @src = `wslpath '#{@src}'`.chomp # Try again with a Windows path
      unless File.exist? @src
        puts "Error: directory '#{@src}' does not exist.".red
        exit 1
      end
    end
    @src += '/outputs/stems' if File.exist?("#{@src}/outputs/stems")

    @dest = if dest.nil? || dest.empty?
              "/var/sitesUbuntu/www.mslinn.com/collections/_songs/stems"
            else
              dest = expand_env dest
              dest.exist? ? dest : `wslpath '#{dest}'`.chomp
            end
    unless File.exist? @dest
      puts "Error: directory #{@dest} does not exist.".red
      exit 2
    end

    @input_filenames =
      Dir
        .glob("#{@src}/*")
        .select { |f| File.file?(f) && !f.end_with?('.zip') }
        .map { |f| File.basename f }
    if @input_filenames.empty?
      puts "There are no audio stems in #{@src}, so this program has nothing to do.".red
      exit
    end

    zipname = common_prefix(@input_filenames)
                .gsub('_stem', '_stems')
                .tr(' ', '_')
                .delete_suffix('_')
    @zip_path = "#{@dest}/#{zipname}.zip"
  end

  def expand_env(str)
    str.gsub(/\$([a-zA-Z_][a-zA-Z0-9_]*)|\${\g<1>}|%\g<1>%/) do
      ENV.fetch(Regexp.last_match(1), nil)
    end
  end

  def make
    FileUtils.rm_f(@zip_path)
    Zip::File.open(@zip_path, create: true) do |zipfile|
      ::Zip.sort_entries = true
      @input_filenames.each do |filename|
        zipfile.add(filename, File.join(@src, filename))
      end
    end
    puts "Wrote #{@zip_path}".green
  end

  def delete_originals
    @input_filenames.each { |f| File.delete "#{@src}/#{f}" }
  end

  def common_prefix(strings)
    return '' if strings.empty?

    result = 0
    (0...strings.first.length).each do |k|
      all_matched = true
      character = strings.first[k]
      strings.each { |str| all_matched &= (character == str[k]) }
      break unless all_matched

      result += 1
    end
    strings.first.slice(0, result)
  end
end

if __FILE__ == $PROGRAM_NAME
  stems = SongStems.new(ARGV[0])
  stems.make
  stems.delete_originals
end

Linked Mp3s and Images

I wrote about cross-file system linking options in NTFS/ext4 Compatible Aliases.

The following script performs the functionality denoted by the solid line with the label symlink in the production overview image. Step by step:

Automatically categories any media file passed to it.
Categorization is done on the basis of file type.
Files can be accepted with syntax compatible with Bash, or using NTFS directory and file naming conventions.
Incoming files are linked to their proper place.

Example Usage

Shell

$ cd $one_year_older
$ song_link outputs/mp3s/One_Year_Older_136_bpm_2024-11-02.mp3
Wrote /var/sitesUbuntu/www.mslinn.com/songs/mp3s/One_Year_Older_136_bpm_2024-11-02.mp3

Shell

$ song_link $one_year_older/outputs/mp3s/One\ Year\ Older_55sec_136_bpm_2024-11-26.mp3
Writing /var/sitesUbuntu/www.mslinn.com/collections/_songs/mp3s/One Year Older_55sec_136_bpm_2024-11-26.mp3

Source Code

song_link

#!/usr/bin/env ruby

require 'clipboard'
require 'colorator'

# If this program is run from either of these song directories:
#   $songs/one_year_older/
#   $songs/one_year_older/outputs/mp3s/
# then by default any pre-exising media file in
# the proper subdirectory of $msp/collections/_songs/
#
# The following is true for two equivalent values of the environment variable called songs:
#   export songs=/mnt/c/media/songs
#   export songs='C:\media\songs'
# This invocation:
#   $ cd $songs/one_year_older
#   $ song_link outputs/mp3s/One_Year_Older_136_bpm_2024-11-02.mp3
# Performs the same action as all of the following:
#   $ song_link $songs/one_year_older/outputs/mp3s/One_Year_Older_136_bpm_2024-11-02.mp3
#   $ song_link $songs/one_year_older/outputs/mp3s/One_Year_Older_136_bpm_2024-11-02.mp3 $msp/collections/_songs/mp3s
#   $ song_link C:/media/songs/one_year_older/outputs/mp3s/One_Year_Older_136_bpm_2024-11-02.mp3
#   $ song_link C:/media/songs/one_year_older/outputs/mp3s/One_Year_Older_136_bpm_2024-11-02.mp3 U:/var/sitesUbuntu/www.mslinn.com/collections/_songs/mp3s
#
# @param src [String] Can be either a Windows path or a WSL path; relative or absolute
# @param dest [String] Defaults to common mp3s directory for all songs in mslinn.com
#                      Will be overwritten if it already exists.
class SongLinker
  def initialize(src = nil, dest = nil)
    @msp       = compute_msp
    @src       = compute_src src
    @src_name  = File.basename @src
    @song_name = compute_song_name @src
    @dest_dir  = compute_dest dest
    @dest_path = "#{@dest_dir}/#{@src_name}"
  end

  def compute_dest(dest)
    dest = expand_env(dest)
    dest_www = if dest.nil? || dest.empty?
                 subdir = compute_subdir @src_name
                 "#{@wsl_vol}#{@msp}/collections/_songs/#{subdir}"
               else
                 win_exist?(dest) ? dest : `wslpath -m '#{dest}'`.chomp
               end
    return dest_www if Dir.exist? dest_www

    if File.exist? dest_www
      puts "Error: Destination '#{dest_www}' is a file, not a directory. Aborting.".red
      exit 1
    end

    puts "Error: Windows directory #{@dest_www} does not exist.".red
    exit 2
  end

  def compute_msp
    msp = ENV.fetch 'msp'
    if msp.nil?
      puts "Error: Environment variable msp is undefined".red
      exit 3
    end
    return msp if Dir.exist? msp

    if File.exist? msp
      puts "Error: Environment variable msp points to a file, not a directory: '#{msp}'".red
    else
      puts "Error: Environment variable msp points to a directory that does not exist: '#{msp}'".red
    end
    exit 4
  end

  def compute_song_name(src)
    result = src
              .delete_prefix(expand_env("$songs/"))
              .split('/')
              .first
    containing_dir = expand_env "$songs/#{result}"
    unless File.exist? containing_dir
      puts "Error: Song directory '#{containing_dir}' does not exist".red
      exit 5
    end
    result
  end

  def compute_src(src)
    src = Dir.pwd if src.nil?
    src = File.realpath expand_env src
    if src.empty?
      puts "Error: src file to copy has no value. Aborting.".red
      exit 6
    end

    unless File.exist? src # Did the user provide a WSL path?
      # TODO: wslpath fails if src file to copy does not exist; need to handle this
      src_tx = `wslpath '#{src}'`.chomp # Try again with a Windows path
      unless File.exist? src_tx
        puts "Error: file to copy '#{src}' does not exist.".red
        exit 7
      end
      return File.realpath src_tx
    end
    File.realpath src
  end

  def compute_subdir(src_name)
    case File.extname(src_name).delete_prefix('.')
    when 'gp'
      'guitar_pro'
    when 'gif', 'jpg', 'jfif', 'png', 'webp'
      'images'
    when 'mp3', 'aiff'
      'mp3s'
    when 'mccz'
      'musescore'
    when 'xml'
      'musicxml'
    when 'pdf'
      'pdfs'
    when 'zip'
      'stems'
    end
  end

  # Does not work
  def display_dir
    win_home_dos = `cmd.exe /c "echo %systemdrive%%homepath%" 2> /dev/null`.chomp
    win_home_linux = `wslpath -au '#{win_home_dos}'`.chomp
    Dir.chdir win_home_linux do
      @wsl_vol = wsl_volume_name
      win_dir = unc_to_win(@dest_path)
      puts "Opening #{win_dir}".yellow
      `cmd.exe /c 'explorer.exe /separate, #{win_dir}'`
    end
  end

  def expand_env(str)
    return nil unless str

    str.gsub(/\$([a-zA-Z_][a-zA-Z0-9_]*)|\${\g<1>}|%\g<1>%/) do
      ENV.fetch(Regexp.last_match(1), nil)
    end
  end

  def link
    if File.exist? @dest_path
      puts "Overwriting #{@dest_path}".green
      File.delete @dest_path
    else
      puts "Writing #{@dest_path}".green
    end
    File.symlink @src, @dest_path
    # Clipboard.copy(@dest_path)
  end

  def unc_to_win(path)
    return path.delete_prefix(@wsl_vol) if path.start_with? @wsl_vol

    return path.delete_prefix('//wsl$/Ubuntu') if path.start_with? '//wsl$/Ubuntu'

    path.delete_prefix('//wsl.localhost/Ubuntu')
  end

  def wsl_volume_name
    line = `cmd.exe /c net.exe use`
            .split("\n")
            .find { |x| x.include? 'Plan 9 Network Provider' }
    if line.nil? || line.empty?
      puts 'WSL has no drive letter mapped. Aborting.'
      exit 1
    end
    line.strip.split.first
  end
end

if __FILE__ == $PROGRAM_NAME
  linker = SongLinker.new(ARGV[0])
  linker.link
  # linker.display_dir
end

Mainframe image; Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License by PekoeBlaze

© Copyright 1994-2024 Michael Slinn. All rights reserved.
For requests to use this copyright-protected work in any manner, email mslinn@mslinn.com.

This website was made using Jekyll and Mike Slinn’s Jekyll Plugins.