PicoRuby Compiler¶ ↑
: author
Monstarlab
: content-source
Ruby Association Grant 2020 Report
: date
July 6, 2021
: allotted-time
30m
: theme
theme
We are¶ ↑
* Monstarlab * Established in 2006 * 85 Million USD of Capital (Nov. 2020) * 1200+ Members in 25 Cities, 16 Countries * Full-Time Remoties in Japan even before the pandemic * Now Hiring! Reach out to me👦
((* *))¶ ↑
# image # src = images/monstarlab.png # relative_width = 110 # relative_margin_top = -3
prop¶ ↑
: hide-title
true
I am¶ ↑
* HASUMI Hitoshi * Shimane development branch * hasumikin@GitHub * hasumikin@Twitter * 🙅hasumin(はすみん) * 🙆hasumikin(はすみきん) # image # src = images/hasumi.jpg # align = right # relative-height = 70 # relative_margin_left = 0
chapter¶ ↑
(('tag:center')) nnn (('tag:xx-large:PicoRuby Compiler'))nn (('tag:xx-large:Why? What?'))
prop¶ ↑
: hide-title
true
: background-image
images/gray-background.png
: background-image-relative-width
100
: background-image-relative-height
100
Ruby for embedded system¶ ↑
# image # src = images/architecture_2.png # relative_width = 100 # relative_margin_top = 0
Ruby for embedded system¶ ↑
# image # src = images/architecture_3.png # relative_width = 100 # relative_margin_top = 0
(('tag:center')) Doesn't make sense 🤔
Why doesn't it make sense?¶ ↑
-
mruby-compiler depends on mruby (mrb_state)
-
You can use mruby-compiler with mruby/c VM
-
However, the size of mruby-compiler spoils mruby/c's small footprint
-
OK
Ruby for embedded system¶ ↑
# image # src = images/architecture_3.png # relative_width = 100 # relative_margin_top = 0
(('tag:center')) Doesn't make sense 🤔
Ruby for embedded system¶ ↑
# image # src = images/architecture_1.png # relative_width = 100 # relative_margin_top = 0
(('tag:center')) PicoRuby Compiler for one-chip microcontrollers 💪
Possibilities of PicoRuby Compiler¶ ↑
* REPL * Debugging on an actual device * Even with mruby VM * One stop solution with mruby/c VM * For newbies, education, Smalruby etc. # image # src = images/hatti.png # align = right # relative-height = 70 # relative_margin_left = 10
(('tag:right'))nn (('tag:xx-small:Copyright © Hikari Arakawa (Ruby Programming Shounendan and Smalruby) 2013'))
chapter¶ ↑
(('tag:center')) nn (('tag:xx-large:PicoRuby Compiler'))nn (('tag:xx-large:History'))nn (('tag:large:July 2019 -'))
prop¶ ↑
: hide-title
true
: background-image
images/gray-background.png
: background-image-relative-width
100
: background-image-relative-height
100
((* *))¶ ↑
prop¶ ↑
: hide-title
true
: background-image
images/osaka.jpg
: background-image-relative-width
100
((* *))¶ ↑
(('tag:center')) nn “Lemon would generate smaller binary than Bison.” nn
prop¶ ↑
: hide-title
true
: background-image
images/matz.jpg
: background-image-relative-width
20
((* *))¶ ↑
# image # src = images/ToyamaRubyKaigi01.png # relative_width = 90 # relative_margin_top = -2
prop¶ ↑
: hide-title
true
github.com/hasumikin/mmrbc.gem¶ ↑
-
mini mruby compiler
-
Tokenizer and Generator
-
CRuby
-
-
Parser
-
C (Lemon) + ffi gem
-
-
Able to compile only `puts “Hello World!”`
-
`[Identifier] “[String literal]”`
-
chapter¶ ↑
(('tag:center')) nnn (('tag:large:Jan 2020 -'))nn (('tag:large:Writing `mmruby` in C'))nn (('tag:large:then, `mv mmruby picoruby`'))
prop¶ ↑
: hide-title
true
: background-image
images/gray-background.png
: background-image-relative-width
100
: background-image-relative-height
100
chapter¶ ↑
(('tag:center')) nnn (('tag:xx-large:RA Grant'))nn (('tag:xx-large:Outcome'))
prop¶ ↑
: hide-title
true
: background-image
images/gray-background.png
: background-image-relative-width
100
: background-image-relative-height
100
Goals¶ ↑
-
Sufficient portion of the Ruby syntax to write IoT firmware
-
Reduce RAM usage to execute on microcontrollers
Syntax - before RA Grant¶ ↑
# enscript ruby # larger_script.rb ary = Array.new(3) ary[0] = {a: 123e2} ary[0][:key] = "string" ary[1] = %w(abc ABC Hello) ary[2] = 0x1f ary[3] = !true puts "my name is #{self.class}, result is #{ary[2] * ary[0][:a]}" p ary => my name is Object, result is 381300 [{:a=>12300, :key=>"string"}, ["abc", "ABC", "Hello"], 31, false]
(('tag:center')) class, def, return, block, if/unless, case-when,n while/until, break, etc were unimplemented yet
Syntax - after RA Grant¶ ↑
# image # src = images/keyboard.rb.png # relative_width = 100
RAM consumption¶ ↑
$ valgrind \ --tool=massif \ --stacks=yes \ path/to/(mrbc|picorbc) \ source.rb $ ms_print massif.out.xxx | less
RAM consumption¶ ↑
---------------------------------------------------------------------------------- Command: ../picoruby/build/host-production/bin/picorbc larger_script.rb Massif arguments: --stacks=yes ms_print arguments: massif.out.13414 ---------------------------------------------------------------------------------- KB 53.20^ # | # | @# ::: | @# : : | @# : : | @@# : : | @@# :: : | @@#::: : | @@#::: : | @@::@@#::: : | @@: @@#::: : | @@:@:::@::::@@: @@#::: :: | @::@::@ :@:: @:: :@@: @@#::: :: | @:::@@@:@::@: @ :@:: @:: :@@: @@#::: :: | @@:@::@:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | ::@@@ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | :::::: :::::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | :::::@@:: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: 0 +----------------------------------------------------------------------->Mi 0 1.616
RAM consumption¶ ↑
# before Grant after Grant mrbc picorbc mrbc(3.0.0) picorbc ================================================================== puts "Hello World!" 157.8 KB 11.17 KB 133.6 KB 15.43 KB ------------------------------------------------------------------ larger_script.rb 162.9 KB 53.20 KB 135.2 KB 24.21 KB
-
Note:
-
using glibc and Linux file system
-
on 64-bit architecture
-
All figures should possibly be about 35% smaller on 32-bit
-
-
RAM consumption¶ ↑
# before Grant after Grant mrbc picorbc mrbc(3.0.0) picorbc ================================================================== puts "Hello World!" 157.8 KB 11.17 KB 133.6 KB 15.43 KB ------------------------------------------------------------------ larger_script.rb 162.9 KB 53.20 KB 135.2 KB 24.21 KB
-
picorbc made a great progress on larger_script.rb 🎉
-
53.20 KB -> 24.21 KB
-
-
but got worse on `puts “Hello World!”`
-
11.17 KB -> 15.43 KB …🤔What happened?
-
chapter¶ ↑
(('tag:center')) nnn (('tag:xx-large:PicoRuby Compiler'))nn (('tag:xx-large:What happened?'))
prop¶ ↑
: hide-title
true
: background-image
images/gray-background.png
: background-image-relative-width
100
: background-image-relative-height
100
What happened?¶ ↑
-
Reusing literal data as much as possible
-
-
Considering paddings & Pooled allocation
-
-
Freeing in loop instead of recursion
-
Reusing literal data as much as possible¶ ↑
-
There were many wasteful, duplicated memory allocations of literal data
-
eg) Reallocates memory when Tokenizer gives data to Parser
-
-
Refactored the code to keep data common in a compilation cycle as much as possible
-
eg) Token data puts on memory should be reused as it is until exactly before generating VM code
Reusing literal data as much as possible¶ ↑
# enscript ruby array[0] = :data
-
Tokens
-
`array`, `[`, `0`, `]`, `=`, `:`, `data`
-
-
Method name should be `[]=`, but where? how?
-
Thus, some literal data should be exceptional
-
Kind of a strategic decision
-
Beware of memory leaks
-
Padding in a struct¶ ↑
# enscript c struct LinkedList { struct LinkedList *next; // 4 bytes (in 32-bit architecture) uint8_t value; // 1 byte } sizeof(LinkedList); => 5 🙅 => 8 🙆
(('tag:center')) ✔Data structure alignment (at least in C99)n pointer + uint8_t + ((padding[3])) = sumn 👉You need to pack them well
Pooled allocation¶ ↑
# enscript c LinkedList *top; (...) top->next->next->next->next; // 5 items in the list
-
Consumes 40 bytes to store 5 values of uint8_t
-
Total size of pointers will be 1 KB if there are 250 items in a list
(('tag:center')) nPooled allocation 💡
Pooled allocation¶ ↑
# enscript c typedef struct node_pool { NodePool *next; uint16_t size; uint16_t index; Node *nodes; } NodePool; NodePool *newNodePool() { size_t size = sizeof(NodePool) + sizeof(Node) * POOL_SIZE; NodePool *node_pool = (NodePool *)alloc(size); memset(node_pool, 0, size); node_pool->next = NULL; node_pool->size = POOL_SIZE; node_pool->index = 0; return node_pool; } /* node_pool.nodes[index] */ (Node *)((Node *)(&node_pool->nodes) + node_pool->index);
Pooled allocation¶ ↑
# enscript bash [n]: bytes *next[4] -----------> *next[4] -----------> *next[4] ------------> size[2] size[2] size[2] index[2] index[2] index[2] node[sizeof(Node)] node[sizeof(Node)] node[sizeof(Node)] node[sizeof(Node)] node[sizeof(Node)] node[sizeof(Node)] ... ... ... node[sizeof(Node)] node[sizeof(Node)] node[sizeof(Node)]
-
Reduces paddings of data structure alignment
-
Reduces the number of pointers
-
Reduces fragmentation
Stack spike¶ ↑
(('tag:center')) Before RA Grant
-------------------------------------------------------------------------------- Command: ../picoruby/build/host-production/bin/picorbc larger_script.rb Massif arguments: --stacks=yes ms_print arguments: massif.out.13414 -------------------------------------------------------------------------------- KB 53.20^ # | # | @# ::: | @# : : | @# : : | @@# : : | @@# :: : | @@#::: : | @@#::: : | @@::@@#::: : | @@: @@#::: : | @@:@:::@::::@@: @@#::: :: | @::@::@ :@:: @:: :@@: @@#::: :: | @:::@@@:@::@: @ :@:: @:: :@@: @@#::: :: | @@:@::@:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | ::@@@ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | :::::: :::::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | :::::@@:: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: 0 +----------------------------------------------------------------------->Mi 0 1.616
Stack spike¶ ↑
# image # src = images/freeTree_0.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_1.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_2.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_3.png # relative_width = 100
Stack spike¶ ↑
(('tag:center')) Before RA Grant
-------------------------------------------------------------------------------- Command: ../picoruby/build/host-production/bin/picorbc larger_script.rb Massif arguments: --stacks=yes ms_print arguments: massif.out.13414 -------------------------------------------------------------------------------- KB 53.20^ # | # | @# ::: | @# : : | @# : : | @@# : : | @@# :: : | @@#::: : | @@#::: : | @@::@@#::: : | @@: @@#::: : | @@:@:::@::::@@: @@#::: :: | @::@::@ :@:: @:: :@@: @@#::: :: | @:::@@@:@::@: @ :@:: @:: :@@: @@#::: :: | @@:@::@:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | ::@@@ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | :::::: :::::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | :::::@@:: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: 0 +----------------------------------------------------------------------->Mi 0 1.616
Stack spike¶ ↑
# image # src = images/freeTree_4.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_5.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_6.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_7.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_8.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_9.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_a.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_b.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_c.png # relative_width = 100
Stack spike¶ ↑
# image # src = images/freeTree_d.png # relative_width = 100
Freeing in loop instead of recursion¶ ↑
# enscript c // Recursion is elegant but a big eater void freeList_in_recursion(List *list) { if (!list) return; freeList_in_recursion(list->next); free(list); } // Loop is somehow clumsy but thrifty void freeList_in_loop(List *list) { List *next; while (list) { next = list->next; free(list); list = next; } }
Freeing in loop instead of recursion¶ ↑
(('tag:center')) before Grant
-------------------------------------------------------------------------------- Command: ../picoruby/build/host-production/bin/picorbc larger_script.rb Massif arguments: --stacks=yes ms_print arguments: massif.out.13414 -------------------------------------------------------------------------------- KB 53.20^ # | # | @# ::: | @# : : | @# : : | @@# : : | @@# :: : | @@#::: : | @@#::: : | @@::@@#::: : | @@: @@#::: : | @@:@:::@::::@@: @@#::: :: | @::@::@ :@:: @:: :@@: @@#::: :: | @:::@@@:@::@: @ :@:: @:: :@@: @@#::: :: | @@:@::@:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | ::@@@ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | :::::: :::::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | :::::@@:: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: | : :: @ :: :::::: ::@ @ :@: @:: @@ :@::@: @ :@:: @:: :@@: @@#::: :: 0 +----------------------------------------------------------------------->Mi 0 1.616
Freeing in loop instead of recursion¶ ↑
(('tag:center')) after Grant
-------------------------------------------------------------------------------- Command: ./build/host-production/bin/picorbc test/fixtures/larger_script.rb Massif arguments: --stacks=yes ms_print arguments: massif.out.3082 -------------------------------------------------------------------------------- KB 24.21^ # | @ @:@:@:# | @:@@@:@:@:@:#: @ | @@:::@:@@@:@:@:@:#::::::@ | @ :::@:@@@:@:@:@:#::::::@ | @ :::@:@@@:@:@:@:#::::::@ | @ :::@:@@@:@:@:@:#::::::@ | @ :::@:@@@:@:@:@:#::::::@ | @@:::::::::::::::@::::: ::::@::@::::::@ :::@:@@@:@:@:@:#::::::@ | @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@ | @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@ | @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@ | @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@ | @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@ | @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@ | @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@ | @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@ | @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@ | @ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@ | ::::@ ::::::: :::: : @:: ::@:: :@::@:: :: @ :::@:@@@:@:@:@:#::::::@ 0 +----------------------------------------------------------------------->Mi 0 1.082
What happened?¶ ↑
-
RAM usage when compiling a very small statement like `puts “Hello World!”` became bigger than before due to Pooled allocation
-
Because Pooled allocation allocates a fixed size of array of struct in advance
-
-
However, when a Ruby script becomes larger, measures of reducing RAM consumption including Pooled allocation work effectively
chapter¶ ↑
(('tag:center')) nnn (('tag:xx-large:PicoRuby Compiler'))nn (('tag:xx-large:Future work'))
prop¶ ↑
: hide-title
true
: background-image
images/gray-background.png
: background-image-relative-width
100
: background-image-relative-height
100
hide-title¶ ↑
(('tag:center')) nnnn (('tag:xx-large:Killer Application'))
prop¶ ↑
: hide-title
true
hide-title¶ ↑
(('tag:center')) nnnn (('tag:x-large:Programming languages need'))n (('tag:x-large:a killer application'))
prop¶ ↑
: hide-title
true
hide-title¶ ↑
(('tag:center')) n (('tag:xx-large:C … UNIX'))nn (('tag:xx-large:PHP … WordPress'))nn (('tag:xx-large:CRuby … Rails'))
prop¶ ↑
: hide-title
true
hide-title¶ ↑
(('tag:center')) nnnn (('tag:xx-large:How about PicoRuby?'))
prop¶ ↑
: hide-title
true
((* *))¶ ↑
# image # src = images/helix_rev3.jpg # relative_width = 107 # relative_margin_top = 1
prop¶ ↑
: hide-title
true
hide-title¶ ↑
(('tag:center'))nn (('tag:xx-large:To be continued on RubyKaigi Takeout 2021 or RubyConf 2021'))n(('🤞if proposals.any?(&:accepted?)🤞'))
prop¶ ↑
: hide-title
true
: background-image
images/yellow-background.png
: background-image-relative-width
100
: background-image-relative-height
100
hide-title¶ ↑
(('tag:center'))nnnn (('tag:xx-large:Thank you!'))nnnn
# image # src = images/Monstarlab_Logo_Grey_CMYK.png # relative_width = 30 # relative_margin_top = 1
prop¶ ↑
: hide-title
true
: background-image
images/yellow-background.png
: background-image-relative-width
100
: background-image-relative-height
100