BNC User Reference Guide

BNC User Reference Guide

BNC User Reference Guide

BNC User Reference Guide

<tokenize>

<tokenize>

<tokenize>

<tokenize>

when tokenization is performed by XAIRA rather than by explicit XML markup. when tokenization is performed by XAIRA rather than by explicit XML markup. when tokenization is performed by XAIRA rather than by explicit XML markup. element tokenize { text } element tokenize { text } element tokenize { text }

tokenize	tokenize	tokenize	tokenize	supplies any additional ICU-conformant rules to be used		supplies any additional ICU-conformant rules to be used		supplies any additional ICU-conformant rules to be used		supplies any additional ICU-conformant rules to be used when tokenization is performed by XAIRA rather than by explicit XML markup.
Declaration	Declaration	Declaration	Declaration							element tokenize { text }
Attributes	Attributes	Attributes	Attributes	(Global attributes only)	(Global attributes only)	(Global attributes only)	(Global attributes only)
Module	Module	Module	Module	module-from-bncxml		module-from-bncxml		module-from-bncxml		module-from-bncxml