Newbie 1.0.7 User Guide

Welcome to Newbie! A simple command-line tool for file processing, text manipulation, and automation.

Newbie is a rapid text processor, designed to not explode in memory even when processing very large files. It is also scripting language designed to be easy to learn and use. Every command starts with `&`, and you can chain them together to do powerful things with your files. The reason all the keywords begin with & is because Newbie has no string delimiters or escape characters. In order to achieve this, Newbie has to parse differently, out to the next &keyword instead of to the next whitespace or quotation mark. This difference makes it exceptionally powerful for text pre-processing. You can do rapid complex bulk text modification, without using regex, sed or grep. If you don't know what I'm talking about, don't worry about it. You won't have to know these things in order to use Newbie. If you do know what I'm talking about, you will appreciate not having any string delimiters or escape characters. When those old Unix tools were first developed, they used them to modify their code, as well as their data. Almost no one edits code that way, anymore, therefore newbie does not require escape characters.

In Newbie you explicitly state what gets displayed on the screen with the &show command. This command is always leftmost, and so may be deleted to enable silent execution in scripts. When the &show command is used the user is assumed to be human, and the output is paginated, etc.

Newbie features a readable pattern language, which is documented in the section on the &find command. This is a very critical feature, and I considered moving it up in the documentation, but it makes sense where it is.

Newbie runs at a slightly lower priority than the user interface components, which does somewhat slow it, but it enables the critical feature of being able to lock your desktop and walk away while processing large files. Otherwise, Newbie would hog the disk and your system would crash when you attempt to unlock the desktop. 

GETTING STARTED

To start in interactive mode (type commands one at a time):
	newbie

Run a script (execute a .ns file):
	newbie myscript.ns

Display this user guide:
	&guide

Exit Newbie:
	&exit

Display the license:
	&license

BASIC FILE OPERATIONS

Show a file:
	&show myfile.txt

Show just the first 10 lines. Note that Newbie generally deals with lines rather than characters. This is part of how it works without string delimiters. The end of a line and &keyword are significant boundaries to the Newbie parser. By default, Newbie truncates long lines at the width of your terminal. The &wrap command toggles this behavior.

	&show myfile.txt &first 10

Show the last 5 lines, this can take a little while on enormous files, but Newbie can handle it:
	&show myfile.txt &last 5

List files in current directory:
	&show &files

List files in a directory:
	&show &files ~/Documents/

List files matching a wildcard pattern:
	&show &files *.txt

Get a file's size in bytes (for use with &math):
	&v.size &= &files &size myfile.txt
	&show &v.size

Copy a file (&copy is really just a passthrough to the rsync -a command):
	&copy source.txt &to destination.txt

Move or rename a file:
	&move oldname.txt &to newname.txt

Delete a file or directory:
	&delete unwanted.txt
	&delete old_directory/

Create a file or directory. A trailing slash indicates directory:
	&create newfile.txt
	&create ~/path/to/new/directory/

Convert file formats. After testing I found the standalone Linux components much faster than the rust crates, and so Newbie uses these, as opposed to native rust. Note that the following file compression formats are transparent in Newbie:
	&convert myfile.txt &into myfile.bz2
	&convert myfile.bz2 &into myfile.gz
	&convert myfile.gz &into myfile.xz
	&convert myfile.xz &into myfile.zst
	&convert myfile.zst &into copiedmyfile.txt

WRITING TO FILES

Write to a file. Note that &write always appends:
	&write Hello, World! &to greeting.txt
	&write Another line &to greeting.txt

Write to the console (stdout) from a script:
	&write Status message &to &display

UNIX PIPELINE INTEGRATION

Newbie can read from stdin and write to stdout using &pipeline, enabling integration with Unix pipes. Scripts must be used (bash interprets & specially on the command line).

Read from stdin:
	&find pattern &in &pipeline &into results.txt

Write to stdout:
	&find pattern &in input.txt &into &pipeline

Filter mode (stdin to stdout):
	&find pattern &in &pipeline &into &pipeline

RUNNING EXTERNAL COMMANDS

Run a bash command:
	&run &bash ls

Run a BASH command and see the output. Note that this will not work with other full screen cli programs like vim or nano. Use another BASH session or a GUI editor:
	&show &run &bash date
	
Run a Newbie script from within a script or interactive session:
	&run myscript.ns

Run a Bash script from within a script or interactive session:
	&run &bash script.sh

VARIABLES

Variables always start with `&v.` and you can use them to store and reuse values:
	&v.name &= Tux Penguin
	&v.age &= 30
	&write Hello, &v.name! You are &v.age years old. &to output.txt

See the value of a variable:
	&show &v.name

See the value of all variables:
	&vars

Clear one variable:
	&v.number &= 1
	&show &v.number
	&empty &v.number
	&show &v.number

Clear multiple variables at once:
	&empty &v.name &v.age

ARITHMETIC AND MATH

Newbie is a text processor, and so treats all characters except EOL and EOF the same. In order for it to use numerical charaters in arithmetic operations the &math keyword must be used:

Variable assignment:
	&v.x &= &math 2 + 2
	&v.area &= &math &v.width * &v.height
	&v.result &= &math (10 + 5) * 2

Supported operators:
	+   Addition
	-   Subtraction
	*   Multiplication
	/   Division
	<   Less than 
	>   Greater than 
	=   Equals 
	()  Parentheses for grouping

Increment and decrement:
	&v.count &= 0
	&v.count++
	&v.count++
	&show &v.count
	&v.count--
	&show &v.count

Using math with conditionals:
	&v.score &= 85
	&if &math &v.score > 60 &write Passed! &to results.txt

Note: &math cannot start a line. Valid forms are:
	&v.x &= &math 2+2
	&if &math &v.x > 3

SYSTEM VARIABLES

Newbie provides some built-in variables, most of these are just the BASH environment variables, along with things that are native to Rust:
	&show &system.user
	&show &system.hostname
	&show &system.path
	&show &system.home
	&show &system.shell
	&show &system.pwd
	&show &system.os
	&show &system.arch
	&show &system.timestamp
	&show &system.time
	&show &system.date
	&show &system.temp
	&write Report generated by &system.user on &system.date &to report.txt

PROCESS VARIABLES

Information about the currently running Newbie process:
	&show &process.pid
	&show &process.ppid
	&show &process.args
	&show &process.argc
	&show &process.cwd

NETWORK VARIABLES

Basic network connectivity information:
	&show &network.connected
	&show &network.local_ip

CONDITIONAL EXECUTION

Newbie uses &if as the sole conditional operator, with test types for different conditions.

Check if a file exists:
	&if &exists myfile.txt &write File exists! &to status.txt

Check if a file does NOT exist:
	&if &not &exists missing.txt &write File not found &to errors.txt

Check if a variable has a value (variable-first syntax):
	&if &v.name &filled &write Name is: &v.name &to log.txt

Check if a variable is empty or undefined (variable-first syntax):
	&if &v.name &empty &write Name not set &to log.txt

Alternate syntax (test-first, also valid):
	&if &filled &v.name &write Name is set &to log.txt
	&if &empty &v.name &write Name not set &to log.txt

Numeric comparisons with &math:
	&v.score &= 75
	&if &math &v.score > 60 &write Passed &to results.txt
	&if &math &v.score < 90 &write Room for improvement &to results.txt
	&if &not &math &v.score > 100 &write Valid score &to results.txt

The test types:
Testing for a file:
	&if &exists Cargo.toml &write Toml exists &to test.txt
Testing if a variable has a value:
	&if &v.name &filled &write Variable has value &to &display
Testing if a variable is empty:
	&if &v.blank &empty &write Variable is empty &to &display
Testing with &math:
	&if &math &v.score < 80 &write Below 80 &to &display

All can be negated with &not:
	&if &not &exists filename
	&if &not &v.varname &filled
	&if &not &math expression

THE &find COMMAND - PATTERN MATCHING

The `&find` command is Newbie's powerful pattern matching tool for searching text in files. Unlike traditional tools like `grep`, `&find` uses an intuitive pattern language that combines literal text, character classes, variables, and positional operators.

Syntax:
	[&show] &find <pattern> &in <file> [&into filename] [options]

IMPORTANT: &find runs silently by default. To see results, use &show:
	&show &find error &in logfile.txt

Or write to a file:
	&find error &in logfile.txt &into matches.txt

Note: Both input and output files can be compressed in any of the formats supported by &convert.

Required components:
	<pattern> - The search pattern (see Pattern Syntax below)
	&in <file> - The file to search (supports compressed files)

Optional components:
	&into filename - Write results to a file instead of displaying
	&first N - Return only the first N matches
	&last N - Return only the last N matches (buffered)
	&numbered - Show line numbers (1, 2, 3...)

PATTERN SYNTAX

Literal Text - Match exact text by typing it directly:
	&find error &in logfile.txt &into &display
	&find Connection refused &in server.log

Character Classes - Match specific types of characters with variable-length or fixed-length patterns.

Variable-length (one or more):
	&numbers      Matches one or more digits (0-9)
	&letters      Matches one or more ASCII letters (a-z, A-Z)
	&text         Matches one or more characters (any UTF-8, non-whitespace)
	&spaces       Matches one or more spaces
	&tabs         Matches one or more tabs

Fixed-length:
	&numbers 4    Matches exactly 4 digits
	&letters 3    Matches exactly 3 ASCII letters
	&text 5       Matches exactly 5 characters (any UTF-8)
	&spaces 2     Matches exactly 2 spaces
	&tabs 1       Matches exactly 1 tab

Note on &letters vs &text:
	&letters matches ASCII letters only (a-z, A-Z)
	&text matches any Unicode characters including international text
	Use &text for multinational data (Chinese, Japanese, Arabic, etc.)

Examples:
	&find Error &numbers &in log.txt
	Match lines containing "Error" followed by digits

	&find ID- &letters 3 &numbers 4 &in data.txt
	Match patterns like "ID-ABC1234"

	&find 名前: &+ &text &in japanese_data.txt
	Match Japanese text after "名前:"

	&find Code: &+ &text 8 &in data.txt
	Match exactly 8 characters (any language) after "Code:"

Positional Operators - Control where patterns match:
	&start &=    Match at beginning of line
	&end &=      Match at end of line

Examples:
	&find &start &= Error &in log.txt
	Match lines beginning with "Error"

	&find &end &= .txt &in listing.txt
	Match lines ending with ".txt"

Anchored Patterns - Use &= to anchor patterns at boundaries:
	&find &start &= Error: &in log.txt
	Match lines starting exactly with "Error:"

	&find &end &= @en . &in wikidata.nt
	Match lines ending exactly with "@en ."

Adjacency Operator - Use &+ to require patterns be immediately adjacent:
	&find &start &= http:// &+ www. &in urls.txt
	Match URLs that start with "http://www." with no space between

	&find < &+ &v.id &+ > &in data.xml
	Match XML tags with captured ID, no spaces inside

CAPTURING DATA WITH &capture

The &capture command extracts data from text into variables using patterns. It is typically used inside a &block for line-by-line processing.

Syntax:
	&capture <pattern with &v.variables> &in <source>

In a block context:
	&block input.txt
	  &capture Name: &v.name &in &newbie.line
	  &if &v.name &filled &write Found: &v.name &to output.txt
	&endblock

Pattern Fences - Use &+ to create boundaries for variable capture:
	&capture ID: &+ &v.id &+ , Name: &+ &v.name &in &newbie.line

This captures:
	- Everything between "ID: " and "," into &v.id
	- Everything after "Name: " to end of match into &v.name

BLOCK PROCESSING

Process files line-by-line with &block:
	&block input.txt
	  &capture pattern &+ &v.data
	  &if &v.data &filled &write &v.data &to output.txt
	  &empty &v.data
	&endblock

Process all files in a directory with &block &files:
	&block &files ~/incoming/
	  &find ERROR
	  &show &newbie.line
	&endblock

This processes every non-hidden file in the directory, iterating through each file's lines sequentially. Subdirectories are skipped. Files are processed in alphabetical order.

Inside a block:
	- &newbie.line contains the current line being processed
	- &capture works on &newbie.line by default
	- &empty should be used to clear variables between iterations

LOOKUP TABLES

Create dictionaries and replace values:
	&lookup dictionary.txt &in input.txt &into output.txt

Dictionary format (tab-separated):
	key1	value1
	key2	value2

All occurrences of keys in the input are replaced with their values.

SORTING

Sort lines in a file:
	&sort input.txt &into sorted.txt

REAL-WORLD EXAMPLES

Example 1: Processing Wikidata RDF (wikidata.ns)

This script extracts and translates data from compressed Wikidata dumps, creating a lookup dictionary and applying it to transform entity IDs into human-readable labels.

&write Started: &+   &+ &system.date &+   &+ &system.time &to &display
&directory ~/testfolder/
&find &end &= @en . &in /mnt/bigdrive/Archive/latest-truthy.nt.bz2 &into enonly.txt
&block enonly.txt
  &empty &v.label &v.entity &v.direct &v.islabel
  &capture <http://www.wikidata.org/entity/ &+ &v.entity &+ > <http://www.w3.org/2000/01/rdf-schema# &+ &v.islabel &+ > " &+ &v.label &+ "@en .
  &capture <http://www.wikidata.org/entity/ &+ &v.entity &+ > <http://www.wikidata.org/prop/direct/ &+ &v.direct &+ > " &+ &v.label &+ "@en .
  &if &v.islabel &filled &write &v.entity &to lookup.txt
  &if &v.islabel &filled &write &v.label &to lookup.txt
  &if &v.direct &filled &write &v.entity &+   &+ &v.direct &+   &+ &v.label &to direct-properties.txt
&endblock
&lookup lookup.txt &in direct-properties.txt &into WDInEnglish.txt
&write Finished: &+   &+ &system.date &+   &+ &system.time &to &display

What this does:
1. Extracts entity IDs and labels using complex patterns from compressed Wikidata English records
2. Builds a lookup dictionary mapping entity IDs to human-readable labels
3. Applies the lookup to replace IDs with labels in the output

Key techniques:
- Working with multi-gigabyte compressed files
- Using &+ for adjacency in complex URI patterns
- Building lookup dictionaries on-the-fly
- Conditional extraction based on RDF predicates

Example 2: Convert RDF to SQL Inserts (tosql.ns)

This script transforms translated Wikidata RDF triples into SQL INSERT statements for database loading. People frequently process bad data in SQL itself, but it's faster to preprocess it with Newbie.

&directory ~/testfolder/
&block enonly.txt
  &empty &v.label &v.entity &v.direct &v.islabel 
  &capture <http://www.wikidata.org/entity/ &+ &v.entity &+ > <http://www.w3.org/2000/01/rdf-schema# &+ &v.islabel &+ > " &+ &v.label &+ "@en .
  &capture <http://www.wikidata.org/entity/ &+ &v.entity &+ > <http://www.wikidata.org/prop/direct/ &+ &v.direct &+ > " &+ &v.label &+ "@en .
  &if &v.islabel &filled &write INSERT INTO entities (entity,label) VALUES (' &+ &v.entity &+ ', ' &+ &v.label &+ '); &to entities.sql
  &if &v.direct &filled &write INSERT INTO properties (entity,direct,label) VALUES (' &+ &v.entity &+ ', ' &+ &v.direct &+ ', ' &+ &v.label &+ '); &to properties.sql
&endblock

What this does:
1. Processes each line of RDF data
2. Captures different triple patterns (labels vs properties)
3. Generates proper SQL INSERT statements with quoted values
4. Separates entity labels and properties into different tables

Key techniques:
- Multiple capture patterns for different triple types
- Variable concatenation with &+ for SQL generation
- Conditional output to multiple files
- Clearing variables between iterations for clean state

Example 3: Unix Pipeline Filter (filter.ns)

A simple filter script demonstrating pipeline integration:

&find pattern &in &pipeline &into &pipeline

Usage from bash:
	cat data.txt | newbie filter.ns | sort > results.txt

Example 4: File Size Guard (process.ns)

Check file size before processing:

&if &not &exists input.txt &write Error: input.txt not found &to &display
&if &exists input.txt &block input.txt
  &v.size &= &files &size input.txt
  &if &math &v.size > 0 &write Processing file of &v.size bytes &to &display
&endblock

Example 5: Comprehensive Test Suite (test.ns)

A complete test suite exercising all Newbie features. Run with: newbie test.ns

The test suite includes tests covering:
- File operations (write, copy, move, delete)
- All variable types (user, system, process, network, global, config)
- Conditionals (&if &exists, &if &filled, &if &empty, &if &math, with &not)
- Arithmetic (&math, increment/decrement)
- Pattern matching (find with literals, character classes, anchors, &text)
- Data extraction (capture with variables and fences)
- Text processing (first, last, numbered output)
- Compression (convert between formats)
- Advanced features (lookup, sort, block processing)
- UTF-8/International text support (&text pattern)
- Pipeline integration (&pipeline, &display)

This test suite serves as both validation and a learning resource showing correct syntax for all commands. It also provides a good example of Newbie comments. Note that any line that does not begin with &keyword is a comment. Whitespace is allowed before the &keyword. There is no comment character.

When complete, the only file remaining in the test directory will be test_results.txt containing all test output.

============================================================================
Newbie Test Script - test.ns
Tests implemented features and provides framework for new features
All output goes to test_results.txt, all temp files are cleaned up
============================================================================

&write ============================================================================ &to test_results.txt
&write Newbie 1.0.7 Test Suite &to test_results.txt
&write Started: &system.date &system.time &to test_results.txt
&write ============================================================================ &to test_results.txt

============================================================================
PHASE 1: Basic File Operations
============================================================================


&write PHASE 1: Basic File Operations &to test_results.txt
&write ---------------------------------------- &to test_results.txt

Test 1: Create test files using &write
&write Line 1 of test file &to _test_file.txt
&write Line 2 of test file &to _test_file.txt
&write Line 3 of test file &to _test_file.txt
&write Line 4 of test file &to _test_file.txt
&write Line 5 of test file &to _test_file.txt
&if &exists _test_file.txt &write Test 1 PASSED: Created test file with &write &to test_results.txt
&if &not &exists _test_file.txt &write Test 1 FAILED: Could not create test file &to test_results.txt

Test 2: Copy a file
&copy _test_file.txt &to _test_copy.txt
&if &exists _test_copy.txt &write Test 2 PASSED: File copied successfully &to test_results.txt
&if &not &exists _test_copy.txt &write Test 2 FAILED: Copy failed &to test_results.txt

Test 3: Move/rename a file
&move _test_copy.txt &to _test_renamed.txt
&if &exists _test_renamed.txt &write Test 3 PASSED: File moved/renamed successfully &to test_results.txt
&if &not &exists _test_renamed.txt &write Test 3 FAILED: Move failed &to test_results.txt

Test 4: Create a directory
&create _testdir/
&if &exists _testdir/ &write Test 4 PASSED: Directory created &to test_results.txt
&if &not &exists _testdir/ &write Test 4 FAILED: Directory creation failed &to test_results.txt

Test 5: Create file in directory
&write Content in subdirectory &to _testdir/subfile.txt
&if &exists _testdir/subfile.txt &write Test 5 PASSED: File created in directory &to test_results.txt
&if &not &exists _testdir/subfile.txt &write Test 5 FAILED: File in directory failed &to test_results.txt

============================================================================
PHASE 2: Variable System
============================================================================


&write PHASE 2: Variable System &to test_results.txt
&write ---------------------------------------- &to test_results.txt

Test 6: Set and check variable
&v.testvar &= Hello from Newbie
&if &v.testvar &filled &write Test 6 PASSED: Variable set successfully: &v.testvar &to test_results.txt
&if &v.testvar &empty &write Test 6 FAILED: Variable not set &to test_results.txt

Test 7: Clear variable
&empty &v.testvar
&if &v.testvar &empty &write Test 7 PASSED: Variable cleared successfully &to test_results.txt
&if &v.testvar &filled &write Test 7 FAILED: Variable not cleared &to test_results.txt

Test 8: System variables
&if &system.user &filled &write Test 8 PASSED: System user is &system.user &to test_results.txt
&if &system.user &empty &write Test 8 FAILED: System user not available &to test_results.txt

Test 9: Process variables
&if &process.pid &filled &write Test 9 PASSED: Process PID is &process.pid &to test_results.txt
&if &process.pid &empty &write Test 9 FAILED: Process PID not available &to test_results.txt

============================================================================
PHASE 3: Arithmetic
============================================================================


&write PHASE 3: Arithmetic &to test_results.txt
&write ---------------------------------------- &to test_results.txt

Test 10: Basic math
&v.x &= &math 2 + 2
&if &math &v.x = 4 &write Test 10 PASSED: 2+2 = &v.x &to test_results.txt
&if &not &math &v.x = 4 &write Test 10 FAILED: 2+2 should equal 4, got &v.x &to test_results.txt

Test 11: Math with variables
&v.a &= 10
&v.b &= 5
&v.c &= &math &v.a * &v.b
&if &math &v.c = 50 &write Test 11 PASSED: 10*5 = &v.c &to test_results.txt
&if &not &math &v.c = 50 &write Test 11 FAILED: 10*5 should equal 50, got &v.c &to test_results.txt

Test 12: Increment
&v.count &= 0
&v.count++
&v.count++
&if &math &v.count = 2 &write Test 12 PASSED: Increment works, count = &v.count &to test_results.txt
&if &not &math &v.count = 2 &write Test 12 FAILED: Increment failed, count = &v.count &to test_results.txt

Test 13: Decrement
&v.count--
&if &math &v.count = 1 &write Test 13 PASSED: Decrement works, count = &v.count &to test_results.txt
&if &not &math &v.count = 1 &write Test 13 FAILED: Decrement failed, count = &v.count &to test_results.txt

Test 14: Parentheses
&v.result &= &math (3 + 2) * 4
&if &math &v.result = 20 &write Test 14 PASSED: (3+2)*4 = &v.result &to test_results.txt
&if &not &math &v.result = 20 &write Test 14 FAILED: (3+2)*4 should equal 20, got &v.result &to test_results.txt

============================================================================
PHASE 4: Conditionals
============================================================================


&write PHASE 4: Conditionals &to test_results.txt
&write ---------------------------------------- &to test_results.txt

Test 15: Check file exists
&if &exists _test_file.txt &write Test 15 PASSED: File exists check &to test_results.txt
&if &not &exists _test_file.txt &write Test 15 FAILED: File exists check &to test_results.txt

Test 16: Check file not exists
&if &not &exists _nonexistent_file.txt &write Test 16 PASSED: File not exists check &to test_results.txt
&if &exists _nonexistent_file.txt &write Test 16 FAILED: File not exists check &to test_results.txt

Test 17: Math conditional greater than
&v.score &= 75
&if &math &v.score > 60 &write Test 17 PASSED: Math greater than (75 > 60) &to test_results.txt
&if &not &math &v.score > 60 &write Test 17 FAILED: Math greater than (75 > 60) &to test_results.txt

Test 18: Math conditional less than
&if &math &v.score < 90 &write Test 18 PASSED: Math less than (75 < 90) &to test_results.txt
&if &not &math &v.score < 90 &write Test 18 FAILED: Math less than (75 < 90) &to test_results.txt

Test 19: Negated math conditional
&if &not &math &v.score > 100 &write Test 19 PASSED: Negated math (NOT 75 > 100) &to test_results.txt
&if &math &v.score > 100 &write Test 19 FAILED: Negated math (NOT 75 > 100) &to test_results.txt

============================================================================
PHASE 5: Pattern Matching
============================================================================


&write PHASE 5: Pattern Matching &to test_results.txt
&write ---------------------------------------- &to test_results.txt

Test 20: Create pattern test file
&write Error 123 occurred &to _pattern_test.txt
&write Warning: low memory &to _pattern_test.txt
&write Error 456 critical failure &to _pattern_test.txt
&write Status OK &to _pattern_test.txt
&write ID-ABC1234 record &to _pattern_test.txt
&if &exists _pattern_test.txt &write Test 20 PASSED: Pattern test file created &to test_results.txt

Test 21: Find literal pattern
&find Error &in _pattern_test.txt &into _find_result.txt
&v.size &= &files &size _find_result.txt
&if &math &v.size > 0 &write Test 21 PASSED: Found literal pattern 'Error' &to test_results.txt
&if &not &math &v.size > 0 &write Test 21 FAILED: Literal pattern not found &to test_results.txt
&delete _find_result.txt

Test 22: Find with &numbers
&find Error &numbers &in _pattern_test.txt &into _find_result.txt
&v.size &= &files &size _find_result.txt
&if &math &v.size > 0 &write Test 22 PASSED: Found pattern with &numbers &to test_results.txt
&if &not &math &v.size > 0 &write Test 22 FAILED: Pattern with &numbers not found &to test_results.txt
&delete _find_result.txt

Test 23: Find with &letters and fixed-length &numbers
&find ID- &letters 3 &+ &numbers 4 &in _pattern_test.txt &into _find_result.txt
&v.size &= &files &size _find_result.txt
&if &math &v.size > 0 &write Test 23 PASSED: Found ID-ABC1234 pattern &to test_results.txt
&if &not &math &v.size > 0 &write Test 23 FAILED: ID pattern not found &to test_results.txt
&delete _find_result.txt

Test 24: Find with &start anchor
&find &start &= Error &in _pattern_test.txt &into _find_result.txt
&v.size &= &files &size _find_result.txt
&if &math &v.size > 0 &write Test 24 PASSED: Found lines starting with Error &to test_results.txt
&if &not &math &v.size > 0 &write Test 24 FAILED: Start anchor pattern not found &to test_results.txt
&delete _find_result.txt

============================================================================
PHASE 6: UTF-8 and International Text (&text)
============================================================================


&write PHASE 6: UTF-8 and International Text &to test_results.txt
&write ---------------------------------------- &to test_results.txt

Test 25: Create UTF-8 test file
&write Name: José García &to _utf8_test.txt
&write 名前: 田中太郎 &to _utf8_test.txt
&write Имя: Иван Петров &to _utf8_test.txt
&write Code: ABCD1234 &to _utf8_test.txt
&if &exists _utf8_test.txt &write Test 25 PASSED: UTF-8 test file created &to test_results.txt

Test 26: Find with &text (variable length)
&find Name: &+ &text &in _utf8_test.txt &into _find_result.txt
&v.size &= &files &size _find_result.txt
&if &math &v.size > 0 &write Test 26 PASSED: Found pattern with &text &to test_results.txt
&if &not &math &v.size > 0 &write Test 26 FAILED: &text pattern not found &to test_results.txt
&delete _find_result.txt

Test 27: Find with &text fixed length
&find Code: &+ &text 8 &in _utf8_test.txt &into _find_result.txt
&v.size &= &files &size _find_result.txt
&if &math &v.size > 0 &write Test 27 PASSED: Found pattern with &text 8 &to test_results.txt
&if &not &math &v.size > 0 &write Test 27 FAILED: Fixed-length &text pattern not found &to test_results.txt
&delete _find_result.txt

============================================================================
PHASE 7: Block Processing
============================================================================


&write PHASE 7: Block Processing &to test_results.txt
&write ---------------------------------------- &to test_results.txt

Test 28: Simple block processing
&v.linecount &= 0
&block _test_file.txt
  &v.linecount++
&endblock
&if &math &v.linecount = 5 &write Test 28 PASSED: Block processed 5 lines &to test_results.txt
&if &not &math &v.linecount = 5 &write Test 28 FAILED: Block linecount = &v.linecount (expected 5) &to test_results.txt

Test 29: Block with capture
&write ID: 12345, Name: TestUser &to _capture_test.txt
&write ID: 67890, Name: AnotherUser &to _capture_test.txt
&v.capturecount &= 0
&block _capture_test.txt
  &empty &v.id &v.name
  &capture ID: &+ &v.id &+ , Name: &+ &v.name
  &if &v.id &filled &v.capturecount++
&endblock
&if &math &v.capturecount = 2 &write Test 29 PASSED: Captured 2 records &to test_results.txt
&if &not &math &v.capturecount = 2 &write Test 29 FAILED: Capture count = &v.capturecount (expected 2) &to test_results.txt

Test 30: Block with &files (multiple files)
&create _multitest/
&write File A content &to _multitest/a.txt
&write File B content &to _multitest/b.txt
&v.filelinecount &= 0
&block &files _multitest/
  &v.filelinecount++
&endblock
&if &math &v.filelinecount = 2 &write Test 30 PASSED: Processed 2 files &to test_results.txt
&if &not &math &v.filelinecount = 2 &write Test 30 FAILED: File line count = &v.filelinecount (expected 2) &to test_results.txt

============================================================================
PHASE 8: Compression
============================================================================


&write PHASE 8: Compression &to test_results.txt
&write ---------------------------------------- &to test_results.txt

Test 31: Convert to gzip
&write Test data for compression testing &to _compress_source.txt
&convert _compress_source.txt &into _compressed.gz
&if &exists _compressed.gz &write Test 31 PASSED: Converted to gzip &to test_results.txt
&if &not &exists _compressed.gz &write Test 31 FAILED: Gzip conversion failed &to test_results.txt

Test 32: Convert gzip to bzip2
&if &exists _compressed.gz &convert _compressed.gz &into _compressed.bz2
&if &exists _compressed.bz2 &write Test 32 PASSED: Converted gzip to bzip2 &to test_results.txt
&if &not &exists _compressed.bz2 &write Test 32 FAILED: Bzip2 conversion failed &to test_results.txt

Test 33: Find in compressed file
&if &exists _compressed.gz &find compression &in _compressed.gz &into _compressed_find.txt
&if &exists _compressed_find.txt &v.size &= &files &size _compressed_find.txt
&if &math &v.size > 0 &write Test 33 PASSED: Found pattern in compressed file &to test_results.txt
&if &not &math &v.size > 0 &write Test 33 FAILED: Pattern not found in compressed file &to test_results.txt

============================================================================
PHASE 9: Lookup Tables
============================================================================


&write PHASE 9: Lookup Tables &to test_results.txt
&write ---------------------------------------- &to test_results.txt

Test 34: Create and use lookup table
&run &bash printf 'apple\tfruit\ncarrot\tvegetable\n' > _lookup_dict.txt
&write I like apple and carrot &to _lookup_input.txt
&lookup _lookup_dict.txt &in _lookup_input.txt &into _lookup_output.txt
&find fruit &in _lookup_output.txt &into _lookup_verify.txt
&v.size &= &files &size _lookup_verify.txt
&if &math &v.size > 0 &write Test 34 PASSED: Lookup table replaced values &to test_results.txt
&if &not &math &v.size > 0 &write Test 34 FAILED: Lookup replacement failed &to test_results.txt

============================================================================
PHASE 10: Sorting
============================================================================


&write PHASE 10: Sorting &to test_results.txt
&write ---------------------------------------- &to test_results.txt

Test 35: Sort file
&write zebra &to _sort_input.txt
&write apple &to _sort_input.txt
&write mango &to _sort_input.txt
&sort _sort_input.txt &into _sort_output.txt
&if &exists _sort_output.txt &write Test 35 PASSED: File sorted successfully &to test_results.txt
&if &not &exists _sort_output.txt &write Test 35 FAILED: Sort failed &to test_results.txt

============================================================================
PHASE 11: File Size
============================================================================


&write PHASE 11: File Size &to test_results.txt
&write ---------------------------------------- &to test_results.txt

Test 36: Get file size
&v.size &= &files &size _test_file.txt
&if &math &v.size > 0 &write Test 36 PASSED: File size is &v.size bytes &to test_results.txt
&if &not &math &v.size > 0 &write Test 36 FAILED: Could not get file size &to test_results.txt

============================================================================
CLEANUP - Remove all temporary files
============================================================================


&write ============================================================================ &to test_results.txt
&write Cleanup: Removing temporary files &to test_results.txt
&write ============================================================================ &to test_results.txt

Cleanup Phase 1 files
&delete _test_file.txt
&delete _test_renamed.txt
&delete _testdir/

Cleanup Phase 5 files
&delete _pattern_test.txt

Cleanup Phase 6 files
&delete _utf8_test.txt

Cleanup Phase 7 files
&delete _capture_test.txt
&delete _multitest/

Cleanup Phase 8 files
&delete _compress_source.txt
&delete _compressed.gz
&delete _compressed.bz2
&delete _compressed_find.txt

Cleanup Phase 9 files
&delete _lookup_dict.txt
&delete _lookup_input.txt
&delete _lookup_output.txt
&delete _lookup_verify.txt

Cleanup Phase 10 files
&delete _sort_input.txt
&delete _sort_output.txt

&write Cleanup complete &to test_results.txt

&write ============================================================================ &to test_results.txt
&write All tests completed: &system.date &system.time &to test_results.txt
&write Results saved to: test_results.txt &to test_results.txt
&write ============================================================================ &to test_results.txt


&write Test suite complete. Results saved to test_results.txt &to &display


Happy scripting!

---

Newbie 1.0.7 - ©2025 Mark Allen Battey - MIT License
