module DataMetaAvro
For command line details either check the new method's source or the README, the usage section.
Constants
- AVRO_TYPES
Mapping from a DataMeta DOM type to a matching renderer of Avro schema JSON. The lambda expect whole DataMetaDom::Field instance, must return the whole specification that you would put under the
"type":
JSON tag, such as:"int"
or, for a type with a size:
{ "type": "fixed", "name": "theFieldName", "size": 16}
Note that wrapping this type into optional specification, i.e. unioned with
"null"
is done by calling the avroType method.- GEM_ROOT
The root of the gem.
- TMPL_ROOT
Location of templates.
- VERSION
Current version
Public Class Methods
# File lib/dataMetaAvro.rb, line 141 def assertMapKeyType(fld, type) raise ArgumentError, %<Field "#{fld.name}": Avro supports only strings as map keys, "#{ type}" is not supported as a map key by Avro> unless type == DataMetaDom::STRING end
Splits the full name of a class into the namespace and the base, returns an array of the namespace (empty string if there is no namespace on the name) and the base name.
Examples:
-
'BaseNameAlone'
->['', 'BaseNameAlone']
-
'one.package.another.pack.FinallyTheName'
->['one.package.another.pack', 'FinallyTheName']
# File lib/dataMetaAvro.rb, line 113 def assertNamespace(fullName) ns, base = DataMetaDom::splitNameSpace(fullName) [DataMetaDom.validNs?(ns, base) ? ns : '', base] end
Converts DataMeta DOM type to Avro schema type.
# File lib/dataMetaAvro.rb, line 73 def avroType(dataMetaType) renderer = AVRO_TYPES[dataMetaType.type] raise "Unsupported type #{dataMetaType}" unless renderer renderer.call(dataMetaType) end
Generates an Avro Schema for the given model's record.
It makes impression that some parameters are not used, but it is not so: they are used by the ERB template as the part of the method's binding.
The parameters nameSpace and the base can be derived from rec, but since they are evaluated previously by calling assertNamespace, can just as well reuse them.
-
Params:
-
model - DataMetaDom::Model
-
outFile - output file name
-
rec - DataMetaDom::Record
-
nameSpace - the namespace for the record
-
base - base name of the record
-
# File lib/dataMetaAvro.rb, line 100 def genRecordJson(model, outFile, rec, nameSpace, base) vars = OpenStruct.new # for template's local variables. ERB does not make them visible to the binding IO.write(outFile, "#{ERB.new(IO.read("#{TMPL_ROOT}/dataClass.avsc.erb"), 0, '-').result(binding)}", {:mode => 'wb'}) end
Generates the Avro Schema, one avsc
file per a record.
# File lib/dataMetaAvro.rb, line 121 def genSchema(model, outRoot) model.records.values.each { |rec| # loop through all the records in the model nameSpace, base = assertNamespace(rec.name) FileUtils.mkdir_p outRoot # write json files named as one.package.another.package.ClassName.json in one dir outFile = File.join(outRoot, "#{rec.name}.avsc") case when rec.kind_of?(DataMetaDom::Record) genRecordJson model, outFile, rec, nameSpace, base else # since we are cycling through records, should never get here raise "Unsupported Entity: #{rec.inspect}" end } end
Shortcut to help for the Hadoop Writables generator.
# File lib/dataMetaAvro.rb, line 136 def helpAvroSchemaGen(file, errorText=nil) DataMetaDom::help(file, "DataMeta DOM Avro Schema Generation ver #{VERSION}", '<DataMeta DOM source> <Avro Schemas target dir>', errorText) end
Wraps required/optional in proper enclosure
# File lib/dataMetaAvro.rb, line 80 def wrapReqOptional(field, baseType) field.isRequired ? baseType : %Q^[#{baseType}, "null"]^ end
Private Instance Methods
# File lib/dataMetaAvro.rb, line 141 def assertMapKeyType(fld, type) raise ArgumentError, %<Field "#{fld.name}": Avro supports only strings as map keys, "#{ type}" is not supported as a map key by Avro> unless type == DataMetaDom::STRING end
Splits the full name of a class into the namespace and the base, returns an array of the namespace (empty string if there is no namespace on the name) and the base name.
Examples:
-
'BaseNameAlone'
->['', 'BaseNameAlone']
-
'one.package.another.pack.FinallyTheName'
->['one.package.another.pack', 'FinallyTheName']
# File lib/dataMetaAvro.rb, line 113 def assertNamespace(fullName) ns, base = DataMetaDom::splitNameSpace(fullName) [DataMetaDom.validNs?(ns, base) ? ns : '', base] end
Converts DataMeta DOM type to Avro schema type.
# File lib/dataMetaAvro.rb, line 73 def avroType(dataMetaType) renderer = AVRO_TYPES[dataMetaType.type] raise "Unsupported type #{dataMetaType}" unless renderer renderer.call(dataMetaType) end
Generates an Avro Schema for the given model's record.
It makes impression that some parameters are not used, but it is not so: they are used by the ERB template as the part of the method's binding.
The parameters nameSpace and the base can be derived from rec, but since they are evaluated previously by calling assertNamespace, can just as well reuse them.
-
Params:
-
model - DataMetaDom::Model
-
outFile - output file name
-
rec - DataMetaDom::Record
-
nameSpace - the namespace for the record
-
base - base name of the record
-
# File lib/dataMetaAvro.rb, line 100 def genRecordJson(model, outFile, rec, nameSpace, base) vars = OpenStruct.new # for template's local variables. ERB does not make them visible to the binding IO.write(outFile, "#{ERB.new(IO.read("#{TMPL_ROOT}/dataClass.avsc.erb"), 0, '-').result(binding)}", {:mode => 'wb'}) end
Generates the Avro Schema, one avsc
file per a record.
# File lib/dataMetaAvro.rb, line 121 def genSchema(model, outRoot) model.records.values.each { |rec| # loop through all the records in the model nameSpace, base = assertNamespace(rec.name) FileUtils.mkdir_p outRoot # write json files named as one.package.another.package.ClassName.json in one dir outFile = File.join(outRoot, "#{rec.name}.avsc") case when rec.kind_of?(DataMetaDom::Record) genRecordJson model, outFile, rec, nameSpace, base else # since we are cycling through records, should never get here raise "Unsupported Entity: #{rec.inspect}" end } end
Shortcut to help for the Hadoop Writables generator.
# File lib/dataMetaAvro.rb, line 136 def helpAvroSchemaGen(file, errorText=nil) DataMetaDom::help(file, "DataMeta DOM Avro Schema Generation ver #{VERSION}", '<DataMeta DOM source> <Avro Schemas target dir>', errorText) end
Wraps required/optional in proper enclosure
# File lib/dataMetaAvro.rb, line 80 def wrapReqOptional(field, baseType) field.isRequired ? baseType : %Q^[#{baseType}, "null"]^ end