module RunnerExecution

Execution primitives that force explicit error handling and never call the shell. Cargo-culted from internal BuildExecution code on top of public version: github.com/square/build_execution

Constants

DEFAULT_LOGGER

Public Class Methods

check_status(cmd, status, output: nil, quiet: false, print_on_failure: false) click to toggle source
# File lib/runner_execution.rb, line 171
def check_status(cmd, status, output: nil, quiet: false, print_on_failure: false)
  return if status.exited? && status.exitstatus == 0

  logger.info(output) if print_on_failure
  # If we exited nonzero or abnormally, print debugging info and explode.
  if status.exited?
    logger.debug("Process Exited normally. Exit status:#{status.exitstatus}") unless quiet
  else
    # This should only get executed if we're stopped or signaled
    logger.debug("Process exited abnormally:\nProcessStatus: #{status.inspect}\n" \
      "Raw POSIX Status: #{status.to_i}\n") unless quiet
  end

  raise RunnerExecutionRuntimeError.new(status, cmd, output)
end
debug_print_cmd_list(cmd_list) click to toggle source
# File lib/runner_execution.rb, line 109
def debug_print_cmd_list(cmd_list)
  # Take a list of command argument lists like you'd sent to open3.pipeline or
  # fail_on_error_pipe and print out a string that would do the same thing when
  # entered at the shell.
  #
  # This is a converter from our internal representation of commands to a subset
  # of bash that can be executed directly.
  #
  # Note this has problems if you specify env or opts
  # TODO: make this remove those command parts
  "\"" +
    cmd_list.map do |cmd|
      cmd.map do |arg|
        arg.gsub("\"", "\\\"") # Escape all double quotes in command arguments
      end.join("\" \"") # Fully quote all command parts, beginning and end.
    end.join("\" | \"") + "\"" # Pipe commands to one another.
end
exit_on_status(output, cmd_list, status_list, quiet: false, print_on_failure: false) click to toggle source

If any of the statuses are bad, exits with the return code of the first one.

Otherwise returns first argument (output)

# File lib/runner_execution.rb, line 160
def exit_on_status(output, cmd_list, status_list, quiet: false, print_on_failure: false)
  status_list.each_index do |index|
    status = status_list[index]
    cmd = cmd_list[index]
    check_status(cmd, status, output: output, quiet: quiet, print_on_failure: print_on_failure)
  end

  output
end
fail_on_error(*cmd, stdin_data: nil, binmode: false, quiet: false, print_on_failure: false, **opts) click to toggle source

Runs a command that fails on error. Uses popen2e wrapper. Handles bad statuses with potential for retries.

# File lib/runner_execution.rb, line 25
def fail_on_error(*cmd, stdin_data: nil, binmode: false, quiet: false, print_on_failure: false, **opts)
  print_command('Running Shell Safe Command:', [cmd]) unless quiet
  shell_safe_cmd = shell_safe(cmd)
  retry_times = opts[:retry] || 0
  opts.delete(:retry)

  while retry_times >= 0
    output, status = popen2e_wrapper(*shell_safe_cmd, stdin_data: stdin_data, binmode: binmode,
                                     quiet: quiet, **opts)

    break unless status.exitstatus != 0

    logger.debug("Command failed with exit status #{status.exitstatus}, retrying #{retry_times} more time(s).") if retry_times > 0
    retry_times -= 1
  end

  # Get out with the status, good or bad.
  # When quiet, we don't need to print the output, as it is already streamed from popen2e_wrapper
  needs_print_on_failure = quiet && print_on_failure
  exit_on_status(output, [shell_safe_cmd], [status], quiet: quiet, print_on_failure: needs_print_on_failure)
end
logger() click to toggle source
# File lib/runner_execution.rb, line 191
def logger
  DEFAULT_LOGGER
end
popen2e_wrapper(*shell_safe_cmd, stdin_data: nil, binmode: false, quiet: false, **opts) click to toggle source

Wrapper around open3.popen2e

We emulate open3.capture2e with the following changes in behavior: 1) The command is printed to stdout before execution. 2) Attempts to use the shell implicitly are blocked. 3) Nonzero return codes result in the process exiting. 4) Combined stdout/stderr goes to callers stdout

(continuously streamed) and is returned as a string

If you’re looking for more process/stream control read the spawn documentation, and pass options directly here

# File lib/runner_execution.rb, line 59
def popen2e_wrapper(*shell_safe_cmd, stdin_data: nil, binmode: false,
                     quiet: false, **opts)

  env = opts.delete(:env) { {} }
  raise ArgumentError, "The :env option must be a hash, not #{env.inspect}" if !env.is_a?(Hash)

  # Most of this is copied from Open3.capture2e in ruby/lib/open3.rb
  _output, _status = Open3.popen2e(env, *shell_safe_cmd, opts) do |i, oe, t|
    if binmode
      i.binmode
      oe.binmode
    end

    outerr_reader = Thread.new do
      if quiet
        oe.read
      else
        # Instead of oe.read, we redirect. Output from command goes to stdout
        # and also is returned for processing if necessary.
        tee(oe, STDOUT)
      end
    end

    if stdin_data
      begin
        i.write stdin_data
      rescue Errno::EPIPE
      end
    end

    i.close
    [outerr_reader.value, t.value]
  end
end
print_command(message, cmd) click to toggle source

Prints a formatted string with command

shell_safe(cmd) click to toggle source

Look at a cmd list intended for spawn. determine if spawn will call the shell implicitly, fail in that case.

# File lib/runner_execution.rb, line 97
def shell_safe(cmd)
  # Take the first string and change it to a list of [executable,argv0]
  # This syntax for calling popen2e (and eventually spawn) avoids
  # the shell in all cases
  shell_safe_cmd = Array.new(cmd)
  if shell_safe_cmd[0].class == String
    shell_safe_cmd[0] = [shell_safe_cmd[0], shell_safe_cmd[0]]
  end
  shell_safe_cmd
end
tee(in_stream, out_stream) click to toggle source

Takes in an input stream and an output stream Redirects data from one to the other until the input stream closes. Returns all data that passed through on return.

# File lib/runner_execution.rb, line 137
def tee(in_stream, out_stream)
  alldata = ''
  loop do
    begin
      data = in_stream.read_nonblock(4096)
      alldata += data
      out_stream.write(data)
      out_stream.flush
    rescue IO::WaitReadable
      IO.select([in_stream])
      retry
    rescue IOError
      break
    end
  end
  alldata
end

Private Instance Methods

check_status(cmd, status, output: nil, quiet: false, print_on_failure: false) click to toggle source
# File lib/runner_execution.rb, line 171
def check_status(cmd, status, output: nil, quiet: false, print_on_failure: false)
  return if status.exited? && status.exitstatus == 0

  logger.info(output) if print_on_failure
  # If we exited nonzero or abnormally, print debugging info and explode.
  if status.exited?
    logger.debug("Process Exited normally. Exit status:#{status.exitstatus}") unless quiet
  else
    # This should only get executed if we're stopped or signaled
    logger.debug("Process exited abnormally:\nProcessStatus: #{status.inspect}\n" \
      "Raw POSIX Status: #{status.to_i}\n") unless quiet
  end

  raise RunnerExecutionRuntimeError.new(status, cmd, output)
end
debug_print_cmd_list(cmd_list) click to toggle source
# File lib/runner_execution.rb, line 109
def debug_print_cmd_list(cmd_list)
  # Take a list of command argument lists like you'd sent to open3.pipeline or
  # fail_on_error_pipe and print out a string that would do the same thing when
  # entered at the shell.
  #
  # This is a converter from our internal representation of commands to a subset
  # of bash that can be executed directly.
  #
  # Note this has problems if you specify env or opts
  # TODO: make this remove those command parts
  "\"" +
    cmd_list.map do |cmd|
      cmd.map do |arg|
        arg.gsub("\"", "\\\"") # Escape all double quotes in command arguments
      end.join("\" \"") # Fully quote all command parts, beginning and end.
    end.join("\" | \"") + "\"" # Pipe commands to one another.
end
exit_on_status(output, cmd_list, status_list, quiet: false, print_on_failure: false) click to toggle source

If any of the statuses are bad, exits with the return code of the first one.

Otherwise returns first argument (output)

# File lib/runner_execution.rb, line 160
def exit_on_status(output, cmd_list, status_list, quiet: false, print_on_failure: false)
  status_list.each_index do |index|
    status = status_list[index]
    cmd = cmd_list[index]
    check_status(cmd, status, output: output, quiet: quiet, print_on_failure: print_on_failure)
  end

  output
end
fail_on_error(*cmd, stdin_data: nil, binmode: false, quiet: false, print_on_failure: false, **opts) click to toggle source

Runs a command that fails on error. Uses popen2e wrapper. Handles bad statuses with potential for retries.

# File lib/runner_execution.rb, line 25
def fail_on_error(*cmd, stdin_data: nil, binmode: false, quiet: false, print_on_failure: false, **opts)
  print_command('Running Shell Safe Command:', [cmd]) unless quiet
  shell_safe_cmd = shell_safe(cmd)
  retry_times = opts[:retry] || 0
  opts.delete(:retry)

  while retry_times >= 0
    output, status = popen2e_wrapper(*shell_safe_cmd, stdin_data: stdin_data, binmode: binmode,
                                     quiet: quiet, **opts)

    break unless status.exitstatus != 0

    logger.debug("Command failed with exit status #{status.exitstatus}, retrying #{retry_times} more time(s).") if retry_times > 0
    retry_times -= 1
  end

  # Get out with the status, good or bad.
  # When quiet, we don't need to print the output, as it is already streamed from popen2e_wrapper
  needs_print_on_failure = quiet && print_on_failure
  exit_on_status(output, [shell_safe_cmd], [status], quiet: quiet, print_on_failure: needs_print_on_failure)
end
logger() click to toggle source
# File lib/runner_execution.rb, line 191
def logger
  DEFAULT_LOGGER
end
popen2e_wrapper(*shell_safe_cmd, stdin_data: nil, binmode: false, quiet: false, **opts) click to toggle source

Wrapper around open3.popen2e

We emulate open3.capture2e with the following changes in behavior: 1) The command is printed to stdout before execution. 2) Attempts to use the shell implicitly are blocked. 3) Nonzero return codes result in the process exiting. 4) Combined stdout/stderr goes to callers stdout

(continuously streamed) and is returned as a string

If you’re looking for more process/stream control read the spawn documentation, and pass options directly here

# File lib/runner_execution.rb, line 59
def popen2e_wrapper(*shell_safe_cmd, stdin_data: nil, binmode: false,
                     quiet: false, **opts)

  env = opts.delete(:env) { {} }
  raise ArgumentError, "The :env option must be a hash, not #{env.inspect}" if !env.is_a?(Hash)

  # Most of this is copied from Open3.capture2e in ruby/lib/open3.rb
  _output, _status = Open3.popen2e(env, *shell_safe_cmd, opts) do |i, oe, t|
    if binmode
      i.binmode
      oe.binmode
    end

    outerr_reader = Thread.new do
      if quiet
        oe.read
      else
        # Instead of oe.read, we redirect. Output from command goes to stdout
        # and also is returned for processing if necessary.
        tee(oe, STDOUT)
      end
    end

    if stdin_data
      begin
        i.write stdin_data
      rescue Errno::EPIPE
      end
    end

    i.close
    [outerr_reader.value, t.value]
  end
end
print_command(message, cmd) click to toggle source

Prints a formatted string with command

shell_safe(cmd) click to toggle source

Look at a cmd list intended for spawn. determine if spawn will call the shell implicitly, fail in that case.

# File lib/runner_execution.rb, line 97
def shell_safe(cmd)
  # Take the first string and change it to a list of [executable,argv0]
  # This syntax for calling popen2e (and eventually spawn) avoids
  # the shell in all cases
  shell_safe_cmd = Array.new(cmd)
  if shell_safe_cmd[0].class == String
    shell_safe_cmd[0] = [shell_safe_cmd[0], shell_safe_cmd[0]]
  end
  shell_safe_cmd
end
tee(in_stream, out_stream) click to toggle source

Takes in an input stream and an output stream Redirects data from one to the other until the input stream closes. Returns all data that passed through on return.

# File lib/runner_execution.rb, line 137
def tee(in_stream, out_stream)
  alldata = ''
  loop do
    begin
      data = in_stream.read_nonblock(4096)
      alldata += data
      out_stream.write(data)
      out_stream.flush
    rescue IO::WaitReadable
      IO.select([in_stream])
      retry
    rescue IOError
      break
    end
  end
  alldata
end