tokenizer

package
v0.0.0-...-5065da5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 15, 2018 License: MIT Imports: 2 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Tokenizer 

type Tokenizer struct{}

func (Tokenizer) Tokenize 

func (_ Tokenizer) Tokenize(src []byte) []string

Tokenize returns slice of tokens. For example, expression like: 

(+ "hello" "world")

Will be converted to: 

"(", "+", "hello", "world", ")"

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.