reduce_network {RWNN} | R Documentation |
Methods for weight and neuron pruning in random weight neural networks.
reduce_network(object, method, retrain = TRUE, ...)
## S3 method for class 'RWNN'
reduce_network(object, method, retrain = TRUE, ...)
## S3 method for class 'ERWNN'
reduce_network(object, method, retrain = TRUE, ...)
object |
An RWNN-object or ERWNN-object. |
method |
A string, or a function, setting the method used to reduce the network (see details). |
retrain |
TRUE/FALSE: Should the output weights be retrained after reduction (defaults to |
... |
Additional arguments passed to the reduction method (see details). |
The 'method
' and additional arguments required by the method are:
"global"
(or "glbl"
)p
: The proportion of weights to remove globally based on magnitude."uniform"
(or "unif"
)p
: The proportion of weights to remove uniformly layer-by-layer based on magnitude."lamp"
p
: The proportion of weights to remove based on LAMP scores."apoz"
p
: The proportion of neurons to remove based on proportion of zeroes produced.tolerance
: The tolerance used when identifying zeroes.type
: A string indicating whether weights should be removed globally ('global'
) or uniformly ('uniform'
)."correlation"
(or "cor"
)type
: The type of correlation (argument passed to cor function).rho
: The correlation threshold used to remove neurons."correlationtest"
(or "cortest"
)type
: The type of correlation (argument passed to cor function).rho
: The correlation threshold used to remove neurons.alpha
: The significance levels used to test whether the observed correlation between two neurons is small than rho
."relief"
p
: The proportion of neurons or weights to remove based on relief scores.type
: A string indicating whether neurons ('neuron'
) or weights ('weight'
) should be removed."output"
tolerance
: The tolerance used when removing zeroes from the output layer.If the object is an ERWNN-object, the reduction is applied to all RWNN-object's in the ERWNN-object. Furthermore, when
the ERWNN-object is created as a stack and the weights of the stack is trained, then 'method
' can be set to:
"stack"
tolerance
: The tolerance used when removing elements from the stack.Lastly, 'method
' can also be passed as a function, with additional arguments passed through the ...
argument.
NB: features and target are passed using the names X
and y
, respectively.
A reduced RWNN-object or ERWNN-object.
Han S., Mao H., Dally W.J. (2016) "Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding." arXiv: 1510.00149.
Hu H., Peng R., Tai Y.W., Tang C.K. (2016) "Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures." arXiv: 1607.03250.
Morcos A.S., Yu H., Paganini M., Tian Y. (2019) "One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers." arXiv: 1906.02773.
Lee J., Park S., Mo S., Ahn S., Shin J. (2021) "Layer-adaptive sparsity for the Magnitude-based Pruning." arXiv: 2010.07611.
Dekhovich A., Tax D.M., Sluiter M.H., Bessa M.A. (2024) "Neural network relief: a pruning algorithm based on neural activity." Machine Learning, 113, 2597-2618.
## RWNN-object
n_hidden <- c(10, 15)
lambda <- 2
m <- rwnn(y ~ ., data = example_data, n_hidden = n_hidden,
lambda = lambda, control = list(lnorm = "l2"))
m |>
reduce_network(method = "relief", p = 0.2, type = "neuron") |>
(\(x) x$weights)()
m |>
reduce_network(method = "relief", p = 0.2, type = "neuron") |>
reduce_network(method = "correlationtest", rho = 0.995, alpha = 0.05) |>
(\(x) x$weights)()
m |>
reduce_network(method = "relief", p = 0.2, type = "neuron") |>
reduce_network(method = "correlationtest", rho = 0.995, alpha = 0.05) |>
reduce_network(method = "lamp", p = 0.2) |>
(\(x) x$weights)()
m |>
reduce_network(method = "relief", p = 0.4, type = "neuron") |>
reduce_network(method = "relief", p = 0.4, type = "weight") |>
reduce_network(method = "output") |>
(\(x) x$weights)()
## ERWNN-object (reduction is performed element-wise on each RWNN)
n_hidden <- c(10, 15)
lambda <- 2
B <- 100
m <- bag_rwnn(y ~ ., data = example_data, n_hidden = n_hidden,
lambda = lambda, B = B, control = list(lnorm = "l2"))
m |>
reduce_network(method = "relief", p = 0.2, type = "neuron") |>
reduce_network(method = "relief", p = 0.2, type = "weight") |>
reduce_network(method = "output")
m <- stack_rwnn(y ~ ., data = example_data, n_hidden = n_hidden,
lambda = lambda, B = B, optimise = TRUE)
# Number of models in stack
length(m$weights)
# Number of models in stack with weights > .Machine$double.eps
length(m$weights[m$weights > .Machine$double.eps])
m |>
reduce_network(method = "stack", tolerance = 1e-8) |>
(\(x) x$weights)()