PicoRuby Compiler¶ ↑
: author
Monstarlab
: content-source
Ruby Association Grant 2020 Report
: date
July 6, 2021
: allotted-time
30m
: theme
theme
We are¶ ↑
* Monstarlab * Established in 2006 * 85 Million USD of Capital (Nov. 2020) * 1200+ Members in 25 Cities, 16 Countries * Full-Time Remoties in Japan even before the pandemic * Now Hiring! Reach out to me👦
((* *))¶ ↑
# image # src = images/monstarlab.png # relative_width = 110 # relative_margin_top = -3
prop¶ ↑
: hide-title
true
I am¶ ↑
* HASUMI Hitoshi * Shimane development branch * hasumikin@GitHub * hasumikin@Twitter * 🙅hasumin（はすみん） * 🙆hasumikin（はすみきん） # image # src = images/hasumi.jpg # align = right # relative-height = 70 # relative_margin_left = 0
chapter¶ ↑
(('tag:center')) nnn (('tag:xx-large:PicoRuby Compiler'))nn (('tag:xx-large:Why? What?'))
prop¶ ↑
: hide-title
true
: background-image
images/gray-background.png
: background-image-relative-width
100
: background-image-relative-height
100
Ruby for embedded system¶ ↑
# image # src = images/architecture_2.png # relative_width = 100 # relative_margin_top = 0
Ruby for embedded system¶ ↑
# image # src = images/architecture_3.png # relative_width = 100 # relative_margin_top = 0
(('tag:center')) Doesn't make sense 🤔
Why doesn't it make sense?¶ ↑
-
mruby-compiler depends on mruby (mrb_state)
-
You can use mruby-compiler with mruby/c VM
-
However, the size of mruby-compiler spoils mruby/c's small footprint
-
OK
Ruby for embedded system¶ ↑
# image # src = images/architecture_3.png # relative_width = 100 # relative_margin_top = 0
(('tag:center')) Doesn't make sense 🤔
Ruby for embedded system¶ ↑
# image # src = images/architecture_1.png # relative_width = 100 # relative_margin_top = 0
(('tag:center')) PicoRuby Compiler for one-chip microcontrollers 💪
Possibilities of PicoRuby Compiler¶ ↑
* REPL * Debugging on an actual device * Even with mruby VM * One stop solution with mruby/c VM * For newbies, education, Smalruby etc. # image # src = images/hatti.png # align = right # relative-height = 70 # relative_margin_left = 10
(('tag:right'))nn (('tag:xx-small:Copyright © Hikari Arakawa (Ruby Programming Shounendan and Smalruby) 2013'))
chapter¶ ↑
(('tag:center')) nn (('tag:xx-large:PicoRuby Compiler'))nn (('tag:xx-large:History'))nn (('tag:large:July 2019 -'))
prop¶ ↑
: hide-title
true
: background-image
images/gray-background.png
: background-image-relative-width
100
: background-image-relative-height
100
((* *))¶ ↑
prop¶ ↑
: hide-title
true
: background-image
images/osaka.jpg
: background-image-relative-width
100
((* *))¶ ↑
(('tag:center')) nn “Lemon would generate smaller binary than Bison.” nn
prop¶ ↑
: hide-title
true
: background-image
images/matz.jpg
: background-image-relative-width
20
((* *))¶ ↑
# image # src = images/ToyamaRubyKaigi01.png # relative_width = 90 # relative_margin_top = -2
prop¶ ↑
: hide-title
true
github.com/hasumikin/mmrbc.gem¶ ↑
-
mini mruby compiler
-
Tokenizer and Generator
-
CRuby
-
-
Parser
-
C (Lemon) + ffi gem
-
-
Able to compile only `puts “Hello World!”`
-
`[Identifier] “[String literal]”`
-
chapter¶ ↑
(('tag:center')) nnn (('tag:large:Jan 2020 -'))nn (('tag:large:Writing `mmruby` in C'))nn (('tag:large:then, `mv mmruby picoruby`'))
prop¶ ↑
: hide-title
true
: background-image
images/gray-background.png
: background-image-relative-width
100
: background-image-relative-height
100
chapter¶ ↑
(('tag:center')) nnn (('tag:xx-large:RA Grant'))nn (('tag:xx-large:Outcome'))
prop¶ ↑
: hide-title
true
: background-image
images/gray-background.png
: background-image-relative-width
100
: background-image-relative-height
100
Goals¶ ↑
-
Sufficient portion of the Ruby syntax to write IoT firmware
-
Reduce RAM usage to execute on microcontrollers
Syntax - before RA Grant¶ ↑
# enscript ruby
# larger_script.rb
ary = Array.new(3)
ary[0] = {a: 123e2}
ary[0][:key] = "string"
ary[1] = %w(abc ABC Hello)
ary[2] = 0x1f
ary[3] = !true
puts "my name is #{self.class}, result is #{ary[2] * ary[0][:a]}"
p ary
=> my name is Object, result is 381300
[{:a=>12300, :key=>"string"}, ["abc", "ABC", "Hello"], 31, false]
(('tag:center')) class, def, return, block, if/unless, case-when,n while/until, break, etc were unimplemented yet
Syntax - after RA Grant¶ ↑
# image # src = images/keyboard.rb.png # relative_width = 100
RAM consumption¶ ↑
$ valgrind \ --tool=massif \ --stacks=yes \ path/to/(mrbc|picorbc) \ source.rb $ ms_print massif.out.xxx | less
RAM consumption¶ ↑
----------------------------------------------------------------------------------
Command: ../picoruby/build/host-production/bin/picorbc larger_script.rb
Massif arguments: --stacks=yes
ms_print arguments: massif.out.13414
----------------------------------------------------------------------------------
KB
53.20^ #
| #
| @# :::
| @# : :
| @# : :
| @@# : :
| @@# :: :
| @@#::: :
| @@#::: :
| @@::@@#::: :
| @@: @@#::: :
| @@:@:::@::::@@: @@#::: ::
| @::@::@ :@:: @:: :@@: @@#::: ::
| @:::@@@:@::@: @ :@:: @:: :@@: @@#::: ::
| @@:@::@:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| ::@@@ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| :::::: :::::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| :::::@@:: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
0 +----------------------------------------------------------------------->Mi
0 1.616
RAM consumption¶ ↑
# before Grant after Grant
mrbc picorbc mrbc(3.0.0) picorbc
==================================================================
puts "Hello World!" 157.8 KB 11.17 KB 133.6 KB 15.43 KB
------------------------------------------------------------------
larger_script.rb 162.9 KB 53.20 KB 135.2 KB 24.21 KB
-
Note:
-
using glibc and Linux file system
-
on 64-bit architecture
-
All figures should possibly be about 35% smaller on 32-bit
-
-
RAM consumption¶ ↑
# before Grant after Grant
mrbc picorbc mrbc(3.0.0) picorbc
==================================================================
puts "Hello World!" 157.8 KB 11.17 KB 133.6 KB 15.43 KB
------------------------------------------------------------------
larger_script.rb 162.9 KB 53.20 KB 135.2 KB 24.21 KB
-
picorbc made a great progress on larger_script.rb 🎉
-
53.20 KB -> 24.21 KB
-
-
but got worse on `puts “Hello World!”`
-
11.17 KB -> 15.43 KB …🤔What happened?
-
chapter¶ ↑
(('tag:center')) nnn (('tag:xx-large:PicoRuby Compiler'))nn (('tag:xx-large:What happened?'))
prop¶ ↑
: hide-title
true
: background-image
images/gray-background.png
: background-image-relative-width
100
: background-image-relative-height
100
What happened?¶ ↑
-
Reusing literal data as much as possible
-
-
Considering paddings & Pooled allocation
-
-
Freeing in loop instead of recursion
-
Reusing literal data as much as possible¶ ↑
-
There were many wasteful, duplicated memory allocations of literal data
-
eg) Reallocates memory when Tokenizer gives data to Parser
-
-
Refactored the code to keep data common in a compilation cycle as much as possible
-
eg) Token data puts on memory should be reused as it is until exactly before generating VM code
Reusing literal data as much as possible¶ ↑
# enscript ruby array[0] = :data
-
Tokens
-
`array`, `[`, `0`, `]`, `=`, `:`, `data`
-
-
Method name should be `[]=`, but where? how?
-
Thus, some literal data should be exceptional
-
Kind of a strategic decision
-
Beware of memory leaks
-
Padding in a struct¶ ↑
# enscript c
struct LinkedList {
struct LinkedList *next; // 4 bytes (in 32-bit architecture)
uint8_t value; // 1 byte
}
sizeof(LinkedList);
=> 5 🙅
=> 8 🙆
(('tag:center')) ✔Data structure alignment (at least in C99)n pointer + uint8_t + ((padding[3])) = sumn 👉You need to pack them well
Pooled allocation¶ ↑
# enscript c LinkedList *top; (...) top->next->next->next->next; // 5 items in the list
-
Consumes 40 bytes to store 5 values of uint8_t
-
Total size of pointers will be 1 KB if there are 250 items in a list
(('tag:center')) nPooled allocation 💡
Pooled allocation¶ ↑
# enscript c
typedef struct node_pool {
NodePool *next;
uint16_t size;
uint16_t index;
Node *nodes;
} NodePool;
NodePool *newNodePool() {
size_t size = sizeof(NodePool) + sizeof(Node) * POOL_SIZE;
NodePool *node_pool = (NodePool *)alloc(size);
memset(node_pool, 0, size);
node_pool->next = NULL;
node_pool->size = POOL_SIZE;
node_pool->index = 0;
return node_pool;
}
/* node_pool.nodes[index] */
(Node *)((Node *)(&node_pool->nodes) + node_pool->index);
Pooled allocation¶ ↑
# enscript bash
[n]: bytes
*next[4] -----------> *next[4] -----------> *next[4] ------------>
size[2] size[2] size[2]
index[2] index[2] index[2]
node[sizeof(Node)] node[sizeof(Node)] node[sizeof(Node)]
node[sizeof(Node)] node[sizeof(Node)] node[sizeof(Node)]
... ... ...
node[sizeof(Node)] node[sizeof(Node)] node[sizeof(Node)]
-
Reduces paddings of data structure alignment
-
Reduces the number of pointers
-
Reduces fragmentation
Stack spike¶ ↑
(('tag:center')) Before RA Grant
--------------------------------------------------------------------------------
Command: ../picoruby/build/host-production/bin/picorbc larger_script.rb
Massif arguments: --stacks=yes
ms_print arguments: massif.out.13414
--------------------------------------------------------------------------------
KB
53.20^ #
| #
| @# :::
| @# : :
| @# : :
| @@# : :
| @@# :: :
| @@#::: :
| @@#::: :
| @@::@@#::: :
| @@: @@#::: :
| @@:@:::@::::@@: @@#::: ::
| @::@::@ :@:: @:: :@@: @@#::: ::
| @:::@@@:@::@: @ :@:: @:: :@@: @@#::: ::
| @@:@::@:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| ::@@@ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| :::::: :::::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| :::::@@:: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
0 +----------------------------------------------------------------------->Mi
0 1.616
Stack spike¶ ↑
# image # src = images/freeTree_0.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_1.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_2.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_3.png # relative_width = 100
Stack spike¶ ↑
(('tag:center')) Before RA Grant
--------------------------------------------------------------------------------
Command: ../picoruby/build/host-production/bin/picorbc larger_script.rb
Massif arguments: --stacks=yes
ms_print arguments: massif.out.13414
--------------------------------------------------------------------------------
KB
53.20^ #
| #
| @# :::
| @# : :
| @# : :
| @@# : :
| @@# :: :
| @@#::: :
| @@#::: :
| @@::@@#::: :
| @@: @@#::: :
| @@:@:::@::::@@: @@#::: ::
| @::@::@ :@:: @:: :@@: @@#::: ::
| @:::@@@:@::@: @ :@:: @:: :@@: @@#::: ::
| @@:@::@:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| ::@@@ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| :::::: :::::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| :::::@@:: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
0 +----------------------------------------------------------------------->Mi
0 1.616
Stack spike¶ ↑
# image # src = images/freeTree_4.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_5.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_6.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_7.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_8.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_9.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_a.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_b.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_c.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_d.png # relative_width = 100
Freeing in loop instead of recursion¶ ↑
# enscript c
// Recursion is elegant but a big eater
void freeList_in_recursion(List *list) {
if (!list) return;
freeList_in_recursion(list->next);
free(list);
}
// Loop is somehow clumsy but thrifty
void freeList_in_loop(List *list) {
List *next;
while (list) {
next = list->next;
free(list);
list = next;
}
}
Freeing in loop instead of recursion¶ ↑
(('tag:center')) before Grant
--------------------------------------------------------------------------------
Command: ../picoruby/build/host-production/bin/picorbc larger_script.rb
Massif arguments: --stacks=yes
ms_print arguments: massif.out.13414
--------------------------------------------------------------------------------
KB
53.20^ #
| #
| @# :::
| @# : :
| @# : :
| @@# : :
| @@# :: :
| @@#::: :
| @@#::: :
| @@::@@#::: :
| @@: @@#::: :
| @@:@:::@::::@@: @@#::: ::
| @::@::@ :@:: @:: :@@: @@#::: ::
| @:::@@@:@::@: @ :@:: @:: :@@: @@#::: ::
| @@:@::@:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| ::@@@ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| :::::: :::::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| :::::@@:: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
| : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: ::
0 +----------------------------------------------------------------------->Mi
0 1.616
Freeing in loop instead of recursion¶ ↑
(('tag:center')) after Grant
--------------------------------------------------------------------------------
Command: ./build/host-production/bin/picorbc test/fixtures/larger_script.rb
Massif arguments: --stacks=yes
ms_print arguments: massif.out.3082
--------------------------------------------------------------------------------
KB
24.21^ #
| @ @:@:@:#
| @:@@@:@:@:@:#: @
| @@:::@:@@@:@:@:@:#::::::@
| @ :::@:@@@:@:@:@:#::::::@
| @ :::@:@@@:@:@:@:#::::::@
| @ :::@:@@@:@:@:@:#::::::@
| @ :::@:@@@:@:@:@:#::::::@
| @@:::::::::::::::@::::: ::::@::@::::::@ :::@:@@@:@:@:@:#::::::@
| @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@
| @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@
| @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@
| @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@
| @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@
| @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@
| @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@
| @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@
| @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@
| @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@
| ::::@ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@
0 +----------------------------------------------------------------------->Mi
0 1.082
What happened?¶ ↑
-
RAM usage when compiling a very small statement like `puts “Hello World!”` became bigger than before due to Pooled allocation
-
Because Pooled allocation allocates a fixed size of array of struct in advance
-
-
However, when a Ruby script becomes larger, measures of reducing RAM consumption including Pooled allocation work effectively
chapter¶ ↑
(('tag:center')) nnn (('tag:xx-large:PicoRuby Compiler'))nn (('tag:xx-large:Future work'))
prop¶ ↑
: hide-title
true
: background-image
images/gray-background.png
: background-image-relative-width
100
: background-image-relative-height
100
hide-title¶ ↑
(('tag:center')) nnnn (('tag:xx-large:Killer Application'))
prop¶ ↑
: hide-title
true
hide-title¶ ↑
(('tag:center')) nnnn (('tag:x-large:Programming languages need'))n (('tag:x-large:a killer application'))
prop¶ ↑
: hide-title
true
hide-title¶ ↑
(('tag:center')) n (('tag:xx-large:C … UNIX'))nn (('tag:xx-large:PHP … WordPress'))nn (('tag:xx-large:CRuby … Rails'))
prop¶ ↑
: hide-title
true
hide-title¶ ↑
(('tag:center')) nnnn (('tag:xx-large:How about PicoRuby?'))
prop¶ ↑
: hide-title
true
((* *))¶ ↑
# image # src = images/helix_rev3.jpg # relative_width = 107 # relative_margin_top = 1
prop¶ ↑
: hide-title
true
hide-title¶ ↑
(('tag:center'))nn (('tag:xx-large:To be continued on RubyKaigi Takeout 2021 or RubyConf 2021'))n(('🤞if proposals.any?(&:accepted?)🤞'))
prop¶ ↑
: hide-title
true
: background-image
images/yellow-background.png
: background-image-relative-width
100
: background-image-relative-height
100
hide-title¶ ↑
(('tag:center'))nnnn (('tag:xx-large:Thank you!'))nnnn
# image # src = images/Monstarlab_Logo_Grey_CMYK.png # relative_width = 30 # relative_margin_top = 1
prop¶ ↑
: hide-title
true
: background-image
images/yellow-background.png
: background-image-relative-width
100
: background-image-relative-height
100