class Subprocess::Process

A child process. The preferred way of spawning a subprocess is through the functions on {Subprocess} (especially {Subprocess::check_call} and {Subprocess::check_output}).

Attributes

command[R]

@return [Array<String>] The command this process was invoked with.

pid[R]

@return [Integer] The process ID of the spawned process.

status[R]

@return [::Process::Status, nil] The exit status code of the process.

Only set after the process has exited.
stderr[R]

@return [IO] The `IO` that is connected to this process's `stderr`.

stdin[R]

@return [IO] The `IO` that is connected to this process's `stdin`.

stdout[R]

@return [IO] The `IO` that is connected to this process's `stdout`.

Public Class Methods

new(cmd, opts={}, &blk) click to toggle source

Create a new process.

@param [Array<String>] cmd The command to run and its arguments (in the

style of an `argv` array). Unlike Python's subprocess module, `cmd`
cannot be a String.

@option opts [IO, Integer, String, Subprocess::PIPE, nil] :stdin The `IO`,

file descriptor number, or file name to use for the process's standard
input. If the magic value {Subprocess::PIPE} is passed, a new pipe will
be opened.

@option opts [IO, Integer, String, Subprocess::PIPE, nil] :stdout The `IO`,

file descriptor number, or file name to use for the process's standard
output. If the magic value {Subprocess::PIPE} is passed, a pipe will be
opened and attached to the process.

@option opts [IO, Integer, String, Subprocess::PIPE, Subprocess::STDOUT,

nil] :stderr The `IO`, file descriptor number, or file name to use for
the process's standard error. If the special value {Subprocess::PIPE} is
passed, a pipe will be opened and attached to the process. If the
special value {Subprocess::STDOUT} is passed, the process's `stderr`
will be redirected to its `stdout` (much like bash's `2>&1`).

@option opts [String] :cwd The directory to change to before executing the

child process.

@option opts [Hash<String, String>] :env The environment to use in the

child process.

@option opts [Array<Integer>] :retain_fds An array of file descriptor

numbers that should not be closed before executing the child process.
Note that, unlike Python (which has :close_fds defaulting to false), all
file descriptors not specified here will be closed.

@option opts [Hash] :exec_opts A hash that will be merged into the options

hash of the call to {::Kernel#exec}.

@option opts [Proc] :preexec_fn A function that will be called in the

child process immediately before executing `cmd`.

@yield [process] Yields the just-spawned {Process} to the optional block.

This occurs after all of {Process}'s error handling has been completed,
and is a great place to call {Process#communicate}, especially when used
in conjunction with {Subprocess::check_call}.

@yieldparam process [Process] The process that was just spawned.

# File lib/subprocess.rb, line 268
def initialize(cmd, opts={}, &blk)
  raise ArgumentError, "cmd must be an Array of strings" unless Array === cmd
  raise ArgumentError, "cmd cannot be empty" if cmd.empty?

  @command = cmd

  # Figure out what file descriptors we should pass on to the child (and
  # make externally visible ourselves)
  @child_stdin, @stdin = parse_fd(opts[:stdin], 'r')
  @child_stdout, @stdout = parse_fd(opts[:stdout], 'w')
  unless opts[:stderr] == STDOUT
    @child_stderr, @stderr = parse_fd(opts[:stderr], 'w')
  end

  retained_fds = Set.new(opts[:retain_fds] || [])

  # A control pipe for ferrying errors back from the child
  control_r, control_w = IO.pipe

  @pid = fork do
    begin
      ::STDIN.reopen(@child_stdin) if @child_stdin
      ::STDOUT.reopen(@child_stdout) if @child_stdout
      if opts[:stderr] == STDOUT
        ::STDERR.reopen(::STDOUT)
      else
        ::STDERR.reopen(@child_stderr) if @child_stderr
      end

      # Set up a new environment if we're requested to do so.
      if opts[:env]
        ENV.clear
        begin
          ENV.update(opts[:env])
        rescue TypeError => e
          raise ArgumentError, "`env` option must be a hash where all keys and values are strings (#{e})"
        end
      end

      # Call the user back, maybe?
      if opts[:preexec_fn]
        if opts[:cwd]
          Dir.chdir(opts[:cwd], &opts[:preexec_fn])
        else
          opts[:preexec_fn].call
        end
      end

      options = {close_others: true}.merge(opts.fetch(:exec_opts, {}))
      if opts[:retain_fds]
        retained_fds.each { |fd| options[fd] = fd }
      end
      if opts[:cwd]
        # We use the chdir option to `exec` since wrapping the
        # `exec` in a Dir.chdir block caused these sporadic errors on macOS:
        # Too many open files - getcwd (Errno::EMFILE)
        options[:chdir] = opts[:cwd]
      end

      begin
        # Ruby's Kernel#exec will call an exec(3) variant if called with two
        # or more arguments, but when called with just a single argument will
        # spawn a subshell with that argument as the command. Since we always
        # want to call exec(3), we use the third exec form, which passes a
        # [cmdname, argv0] array as its first argument and never invokes a
        # subshell.
        exec([cmd[0], cmd[0]], *cmd[1..-1], options)
      rescue TypeError => e
        raise ArgumentError, "cmd must be an Array of strings (#{e})"
      end

    rescue Exception => e
      # Dump all errors up to the parent through the control socket
      Marshal.dump(e, control_w)
      control_w.flush
    end

    # Something has gone terribly, terribly wrong if we're hitting this :(
    exit!(1)
  end

  # Meanwhile, in the parent process...

  # First, let's close some things we shouldn't have access to
  @child_stdin.close if our_fd?(opts[:stdin])
  @child_stdout.close if our_fd?(opts[:stdout])
  @child_stderr.close if our_fd?(opts[:stderr])
  control_w.close

  # Any errors during the spawn process? We'll get past this point when the
  # child execs and the OS closes control_w
  begin
    e = Marshal.load(control_r)
    e = "Unknown Failure" unless e.is_a?(Exception) || e.is_a?(String)
    # Because we're throwing an exception and not returning a
    # Process, we need to make sure the child gets reaped
    wait
    raise e
  rescue EOFError # Nothing to read? Great!
  ensure
    control_r.close
  end

  # Everything is okay. Good job, team!
  blk.call(self) if blk
end

Private Class Methods

catching_sigchld(pid) { |sigchld_global_read, self_read| ... } click to toggle source

@param [Integer] pid @return [void]

# File lib/subprocess.rb, line 687
def self.catching_sigchld(pid)
  IO.pipe do |self_read, self_write|
    begin
      register_pid(pid, self_write)
      yield @sigchld_global_read, self_read
    ensure
      unregister_pid(pid)
    end
  end
end
handle_sigchld() click to toggle source

@return [void]

# File lib/subprocess.rb, line 616
def self.handle_sigchld
  # We'd like to just notify everything in `@sigchld_fds`, but
  # ruby signal handlers are not executed atomically with respect
  # to other Ruby threads, so reading it is racy. We can't grab
  # `@sigchld_mutex`, because signal execution blocks the main
  # thread, and so we'd deadlock if the main thread currently
  # holds it.
  #
  # Instead, we keep a long-lived notify self-pipe that we select
  # on inside `communicate`, and we task `communicate` with
  # grabbing the lock and fanning out the wakeups.
  begin
    @sigchld_global_write.write_nonblock("\x00")
  rescue Errno::EWOULDBLOCK, Errno::EAGAIN
    nil # ignore
  end
end
register_pid(pid, fd) click to toggle source

@param [Integer] pid @param [IO] fd @return [void]

# File lib/subprocess.rb, line 657
def self.register_pid(pid, fd)
  @sigchld_mutex.synchronize do
    @sigchld_fds[pid] = fd
    if @sigchld_fds.length == 1
      if @sigchld_global_write.nil? || @sigchld_pipe_pid != ::Process.pid
        # Check the PID so that if we fork we will re-open the
        # pipe. It's important that a fork parent and child don't
        # share this pipe, because if they do they risk stealing
        # each others' wakeups.
        @sigchld_pipe_pid = ::Process.pid
        @sigchld_global_read, @sigchld_global_write = IO.pipe
      end
      @sigchld_old_handler = Signal.trap('SIGCHLD') {handle_sigchld}
    end
  end
end
unregister_pid(pid) click to toggle source

@param [Integer] pid @return [void]

# File lib/subprocess.rb, line 676
def self.unregister_pid(pid)
  @sigchld_mutex.synchronize do
    if @sigchld_fds.length == 1
      Signal.trap('SIGCHLD', @sigchld_old_handler || 'DEFAULT')
    end
    @sigchld_fds.delete(pid)
  end
end
wakeup_sigchld() click to toggle source

Wake up everyone. We can't tell who we should wake up without `wait`ing, and we want to let the process itself do that. In practice, we're not likely to have that many in-flight subprocesses, so this is probably not a big deal. @return [void]

# File lib/subprocess.rb, line 639
def self.wakeup_sigchld
  @sigchld_mutex.synchronize do
    @sigchld_fds.values.each do |fd|
      begin
        fd.write_nonblock("\x00")
      rescue Errno::EWOULDBLOCK, Errno::EAGAIN
        # If the pipe is full, the other end will be woken up
        # regardless when it next reads, so it's fine to skip the
        # write (the pipe is a wakeup channel, and doesn't contain
        # meaningful data).
      end
    end
  end
end

Public Instance Methods

communicate(input=nil, timeout_s=nil) { |stdout, stderr| ... } click to toggle source

Write the (optional) input to the process's `stdin` and read the contents of `stdout` and `stderr`. If a block is provided, stdout and stderr are yielded as they are read. Otherwise they are buffered in memory and returned when the process exits. Do this all using `IO::select`, so we don't deadlock due to full pipe buffers.

This is only really useful if you set some of `:stdin`, `:stdout`, and `:stderr` to {Subprocess::PIPE}.

@param [String] input A string to feed to the child's standard input. @param [Numeric] timeout_s Raise {Subprocess::CommunicateTimeout} if communicate

does not finish after timeout_s seconds.

@yieldparam [String] stdout Data read from stdout since the last yield @yieldparam [String] stderr Data read from stderr since the last yield @return [Array(String, String), nil] An array of two elements: the data read from the

child's standard output and standard error, respectively.
Returns nil if a block is provided.
# File lib/subprocess.rb, line 428
def communicate(input=nil, timeout_s=nil)
  raise ArgumentError if !input.nil? && @stdin.nil?

  stdout, stderr = "", ""

  # NB: Always force encoding to binary so we handle unicode or binary input
  # correctly across multiple write_nonblock calls, since we manually slice
  # the input depending on how many bytes were written
  input = input.dup.force_encoding('BINARY') unless input.nil?

  @stdin.close if (input.nil? || input.empty?) && !@stdin.nil?

  timeout_at = Time.now + timeout_s if timeout_s

  self.class.catching_sigchld(pid) do |global_read, self_read|
    wait_r = [@stdout, @stderr, self_read, global_read].compact
    wait_w = [input && @stdin].compact
    done = false
    while !done
      # If the process has exited, we want to drain any remaining output before returning
      if poll
        ready_r = wait_r - [self_read, global_read]
        ready_w = []
        done = true
      else
        ready_r, ready_w = select_until(wait_r, wait_w, [], timeout_at)
        raise CommunicateTimeout.new(@command, stdout, stderr) if ready_r.nil?
      end

      if ready_r.include?(@stdout)
        if drain_fd(@stdout, stdout)
          wait_r.delete(@stdout)
        end
      end

      if ready_r.include?(@stderr)
        if drain_fd(@stderr, stderr)
          wait_r.delete(@stderr)
        end
      end

      if ready_r.include?(global_read)
        if drain_fd(global_read)
          raise "Unexpected internal error -- someone closed the global self-pipe!"
        end
        self.class.wakeup_sigchld
      end

      if ready_r.include?(self_read)
        if drain_fd(self_read)
          raise "Unexpected internal error -- someone closed our self-pipe!"
        end
      end

      if ready_w.include?(@stdin)
        written = 0
        begin
          written = @stdin.write_nonblock(input)
        rescue EOFError # Maybe I shouldn't catch this...
        rescue Errno::EINTR
        rescue IO::WaitWritable
          # On OS X, a pipe can raise EAGAIN even after select indicates
          # that it is writable. Once the process consumes from the pipe,
          # the next write should succeed and we should make forward progress.
          # Until then, treat this as not writing any bytes and continue looping.
          # For details see: https://github.com/stripe/subprocess/pull/22
          nil
        rescue Errno::EPIPE
          # The other side of the pipe closed before we could
          # write all of our input. This can happen if the
          # process exits prematurely.
          @stdin.close
          wait_w.delete(@stdin)
        end
        input[0...written] = ''
        if input.empty?
          @stdin.close
          wait_w.delete(@stdin)
        end
      end

      if block_given? && !(stderr.empty? && stdout.empty?)
        yield stdout, stderr
        stdout, stderr = "", ""
      end
    end
  end

  wait

  if block_given?
    nil
  else
    [stdout, stderr]
  end
end
drain_fd(fd, buf=nil) click to toggle source

Do nonblocking reads from `fd`, appending all data read into `buf`.

@param [IO] fd The file to read from. @param [String] buf A buffer to append the read data to.

@return [true, false] Whether `fd` was closed due to an exceptional

condition (`EOFError` or `EPIPE`).
# File lib/subprocess.rb, line 398
def drain_fd(fd, buf=nil)
  loop do
    tmp = fd.read_nonblock(4096).force_encoding(fd.external_encoding)
    buf << tmp unless buf.nil?
  end
rescue EOFError, Errno::EPIPE
  fd.close
  true
rescue Errno::EINTR
rescue Errno::EWOULDBLOCK, Errno::EAGAIN
  false
end
poll() click to toggle source

Poll the child, setting (and returning) its status. If the child has not terminated, return nil and exit immediately

@return [::Process::Status, nil] The exit status of the process

# File lib/subprocess.rb, line 379
def poll
  @status ||= (::Process.waitpid2(@pid, ::Process::WNOHANG) || []).last
end
send_signal(signal) click to toggle source

Does exactly what it says on the box.

@param [String, Symbol, Integer] signal The signal to send to the child

process. Accepts all the same arguments as Ruby's built-in
{::Process::kill}, for instance a string like "INT" or "SIGINT", or a
signal number like 2.

@return [Integer] See {::Process.kill}

@see ::Process.kill

# File lib/subprocess.rb, line 535
def send_signal(signal)
  ::Process.kill(signal, pid)
end
terminate() click to toggle source

Sends `SIGTERM` to the process.

@return [Integer] See {send_signal}

@see send_signal

# File lib/subprocess.rb, line 544
def terminate
  send_signal("TERM")
end
wait() click to toggle source

Wait for the child to return, setting and returning the status of the child.

@return [::Process::Status] The exit status of the process

# File lib/subprocess.rb, line 387
def wait
  @status ||= ::Process.waitpid2(@pid).last
end

Private Instance Methods

our_fd?(fd) click to toggle source

The pair to parse_fd, returns whether or not the file descriptor was opened by us (and therefore should be closed by us).

@param [IO, Integer, String, nil] fd @return [Boolean]

# File lib/subprocess.rb, line 581
def our_fd?(fd)
  case fd
  when PIPE, String
    true
  else
    false
  end
end
parse_fd(fd, mode) click to toggle source

Return a pair of values (child, mine), which are how the given file descriptor should appear to the child and to this process, respectively. “mine” is only non-nil in the case of a pipe (in fact, we just return a list of length one, since ruby will unpack nils from missing list items).

@param [IO, Integer, String, nil] fd @param [String] mode @return [Array<IO>]

# File lib/subprocess.rb, line 557
def parse_fd(fd, mode)
  fds = case fd
  when PIPE
    IO.pipe
  when IO
    [fd]
  when Integer
    [IO.new(fd, mode)]
  when String
    [File.open(fd, mode)]
  when nil
    []
  else
    raise ArgumentError
  end

  mode == 'r' ? fds : fds.reverse
end
select_until(read_array, write_array, err_array, timeout_at) click to toggle source

Call IO.select timing out at Time `timeout_at`. If `timeout_at` is nil, never times out.

@param [Array<IO>, nil] read_array @param [Array<IO>, nil] write_array @param [Array<IO>, nil] err_array @param [Integer, Float, nil] timeout_at @return [Array<Array<IO>>, nil]

# File lib/subprocess.rb, line 597
def select_until(read_array, write_array, err_array, timeout_at)
  if !timeout_at
    return IO.select(read_array, write_array, err_array)
  end

  remaining = (timeout_at - Time.now)
  return nil if remaining <= 0

  IO.select(read_array, write_array, err_array, remaining)
end