salsa  0.4.0
SALSA - Scalable Adaptive Large Structures Analysis

The goal of the project is to develop framework, that will distribute large job between workers and process it in parallel. Framework should be able to adapt to different conditions in order to analyse data as effectively as possible.

Run via docker/podman

Note: One can subsitute command podman by docker

Simple cluster using UDP discovery

Cluster

podman run -it --rm --net=host -e SALSA_PUB_URL=">tcp://localhost:5555" registry.openbrain.sk/salsa/salsa

Start monitoring (optional)

podman run -it --rm --net=host --entrypoint salsa-mon registry.openbrain.sk/salsa/salsa -u "tcp://*:5555"

Submit job

podman run -it --rm --net=host --entrypoint salsa-feeder registry.openbrain.sk/salsa/salsa -u tcp://localhost:41000 -t "sleep 5:10"

Simple cluster using GOSSIP discovery

Discovery

podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n disovery

Redirector

podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n rdr

Worker

podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n wk

Submit job

podman run -it --rm --net=host --entrypoint salsa-feeder registry.openbrain.sk/salsa/salsa -u tcp://localhost:41000 -t "sleep 5:10"

Redirector and worker

podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n rdr+wk

Only redirector with debug message and publisher set

podman run -it --rm --net=host -e SALSA_PUB_URL=">tcp://localhost:5555" registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n rdr --debug

Extra option as cmd parameter

podman run -it --rm --net=host registry.openbrain.sk/salsa/salsa -c /etc/salsa/gossip.yaml -n rdr{ip:192.168.1.20,port:35000}

Installation via yum/dnf

Install OpenBrain repository

yum install yum-plugin-copr -y
yum copr enable obl/stable -y

Install salsa package

yum install salsa -y

Develop

Install all dependencies

yum install redhat-lsb cmake gcc-c++ zyre-devel spdlog-devel protobuf-devel jsoncpp-devel yaml-cpp-devel doxygen ncurses-devel ndm-devel help2man

Compile project

cd <project-dir>
scripts/make.sh

scripts/make.sh options

Option Description
install Install after compilation.
clean Destroy build folder before building.
clang Use Clang compiler instead of system default.
strict Enable all warnings (requires clang). Not -Werror!
ninja Use Ninja build system instead of system default.
doc Build documentation (via Doxygen)
release Build as release. (This is used in pipeline.) Infers doc and strict. Warnings as errors.

strict option was created for development purposes, so you can compile with all warnings, while not failing build (no -Werror).

If you are contributing, make sure that you pass compilation with at least release option. Even better if you pass with release clang (thanks to Clang's -Weverything).

Example running master-worker arch from developemnt

$SALSA_ROOT/build/src/salsa -c $SALSA_ROOT/etc/config.json -n rdr,wk --debug # Runs redirector and 1 worker. Beware! Every worker gets all cores of machine!
$SALSA_ROOT/build/src/salsa-feeder -u <submiter-url> -f <path/to/list> # Command list is one command per line. (<submiter-url> : tcp://localhost:41000)
$SALSA_ROOT/build/src/salsa-feeder -u <submiter-url> -t "cmd arg1 arg2:N" # Using template switch executable is "cmd arg1 arg2" and it is done "N" times

Publishing info about jobs

Run before (on rdr)

export SALSA_PUB_URL=">tcp://localhost:5555"

## Kill job with specific JOB ID

salsa-feeder -u tcp://localhost:41000 -c "JOB_DEL:<job_id>"

Next command will work for JOB_ID= E25633E7362F43FF8DFE32E3C525E223

salsa-feeder -u tcp://localhost:41000 -c "JOB_DEL:E25633E7362F43FF8DFE32E3C525E223"

Kill all jobs

salsa-feeder -u tcp://localhost:41000 -c "JOB_DEL_ALL"