Documentation
¶
Overview ¶
Package archive implements methods for working with archives, including archiving and compressing tar archives using tar and zstd libraries. Additionally, a stream pipeline is defined that enables archival, compression, and digesting operations to occur in a modular fashion with one pass through a file.
Index ¶
- Variables
- func AssignCompressionLevel(userSpecifiedLevel string) zstd.EncoderLevel
- func EncoderWithCustomLevel(lv zstd.EncoderLevel) *zstd.Encoder
- func ExtractTar(ctx context.Context, rc io.ReadCloser, destPath string) error
- func ExtractTarCompat(ctx context.Context, rc io.ReadCloser, destPath string) error
- func ExtractTarZstd(ctx context.Context, rc io.ReadCloser, destPath string) error
- func ExtractTarZstdCompat(ctx context.Context, rc io.ReadCloser, destPath string) error
- func ExtractZstd(ctx context.Context, rc io.ReadCloser, destPath string) error
- func TarToStream(ctx context.Context, fsys fs.FS, ostream io.Writer, ...) error
- type BufWriteCloser
- type FlushWriter
- type PipeCompressThreshold
- type PipeCounter
- func (pc *PipeCounter) AddProgressTracker(pt ProgressFunc) PipeReader
- func (pc *PipeCounter) Close() error
- func (pc *PipeCounter) ConnectIn(r io.ReadCloser) PipeReader
- func (pc *PipeCounter) ConnectOut(w io.WriteCloser) PipeWriter
- func (pc *PipeCounter) Read(p []byte) (n int, err error)
- func (pc *PipeCounter) Write(p []byte) (n int, err error)
- type PipeCreateFile
- type PipeDigest
- func (pd *PipeDigest) Close() error
- func (pd *PipeDigest) ConnectIn(r io.ReadCloser) PipeReader
- func (pd *PipeDigest) ConnectOut(w io.WriteCloser) PipeWriter
- func (pd *PipeDigest) GetDigest() digest.Digest
- func (pd *PipeDigest) Read(p []byte) (n int, err error)
- func (pd *PipeDigest) Write(p []byte) (n int, err error)
- type PipeGzDec
- type PipeGzEnc
- type PipeIn
- type PipeOut
- type PipeReader
- type PipeTerm
- type PipeWriter
- type PipeZstdDec
- type PipeZstdEnc
- type ProgressFunc
- type TarArchiver
- type TarArchiverOption
- type TarExtractor
- type TarFileData
Constants ¶
This section is empty.
Variables ¶
var ErrNonPortablePath = errors.New("non-portable path")
ErrNonPortablePath is used to denote a filename/path that contains invalid characters.
Functions ¶
func AssignCompressionLevel ¶
func AssignCompressionLevel(userSpecifiedLevel string) zstd.EncoderLevel
AssignCompressionLevel Helper function to assign determine custom encoding level.
func EncoderWithCustomLevel ¶
func EncoderWithCustomLevel(lv zstd.EncoderLevel) *zstd.Encoder
EncoderWithCustomLevel returns an Encoder at the specified compression level.
func ExtractTar ¶
ExtractTar unarchives a tar file. Existing files are overwritten.
func ExtractTarCompat ¶
ExtractTarCompat unarchives a tar file. Existing files are overwritten. Parent directories are created as needed.
func ExtractTarZstd ¶
ExtractTarZstd decompresses and unarchives a tar+zst file. Existing files are overwritten.
func ExtractTarZstdCompat ¶
ExtractTarZstdCompat decompresses and unarchives a tar+zst file. Existing files are overwritten. Parent directories are created as needed.
func ExtractZstd ¶
ExtractZstd decompresses a .zst file. Existing files are overwritten.
func TarToStream ¶
func TarToStream(ctx context.Context, fsys fs.FS, ostream io.Writer, options ...TarArchiverOption) error
TarToStream archives a list of files and writes the archive to the provided stream.
Types ¶
type BufWriteCloser ¶
BufWriteCloser is a wrapper around bytes.buffer that provides a no-op close for io.WriteCloser compatibility.
func (*BufWriteCloser) Close ¶
func (bwc *BufWriteCloser) Close() error
Close for BufWriteCloser doesn't need to do anything, just for interface compat.
type FlushWriter ¶
FlushWriter combines closer, writer, and Flush, used for tracking compressors that must flush data before checking the output size.
type PipeCompressThreshold ¶
type PipeCompressThreshold struct {
W io.WriteCloser // final output writer
FinalRatio float32 // if nonzero, this is the resulting ratio (uncompressed/compressed) for reporting purposes
DidCompress bool // true if the compression ratio check passed and the data was compressed.
// contains filtered or unexported fields
}
PipeCompressThreshold performs compression on input and forwards compressed data to the output, as long as the compression ratio meets a specified threshold. If the compression threshold is NOT met, the data is instead piped directly to the output, uncompressed. To perform this, the first N bytes of the input are buffered, then compressed into a second buffer. When the data size reaches N, the size of the two buffers are compared to determine the ratio. If the ratio meets desired threshold, the compressed buffer is forwarded to the output pipe, and incoming data is no longer buffered. Conversely, if the ratio is not met, the uncompressed buffer is forwarded to the output, and the compression process is bypassed for the remainder of the data.
func NewPipeCompressThreshold ¶
func NewPipeCompressThreshold(lvl zstd.EncoderLevel, checkSize int, useGz bool) *PipeCompressThreshold
NewPipeCompressThreshold creates a pipe writer that will buffer the initial checkSize bytes, keeping two versions, a compressed and uncompressed one. When the checkSize is met, the compressibility of the data is examined. If the data is not compressible, then the uncompressed data is output. Further writes after the check will no longer buffer data, and will instead pipe data directly to the output (or compressor). lvl indicates a zstd compression level to use. Currently, levels are only supported for zstd, and lvl is ignored for Gzip. useGz should be set to true to use a gzip encoder versus the zstd one.
func (*PipeCompressThreshold) Close ¶
func (pct *PipeCompressThreshold) Close() error
Close for PipeCompressThreshold will simply forward a close operation down the pipe if the compression check threshold has already been met. If that threshold has not been met, however, we have to flush the compressed or uncompressed buffer to the output based on compression ratio.
func (*PipeCompressThreshold) ConnectOut ¶
func (pct *PipeCompressThreshold) ConnectOut(w io.WriteCloser) PipeWriter
ConnectOut establishes the temporary buffers, and sets the final output for when a compression decision has been reached. Until the threshold is reached, data is multiwritten to two temporary buffered outputs. Due to this buffering, output from PipeCompressThreshold is delayed (the delay goes away once the threshold is met). Ultimately, the data will be either compressed or uncompressed depending on the threshold calculation.
func (*PipeCompressThreshold) IsCompressible ¶
func (pct *PipeCompressThreshold) IsCompressible() bool
IsCompressible returns true if the ratio compressed/uncompressed is less than a threshold. This is safe to call after the write process is complete to determine if the initial data was compressible.
func (*PipeCompressThreshold) Write ¶
func (pct *PipeCompressThreshold) Write(p []byte) (n int, err error)
Write for PipeCompressThreshold writes incoming data to two output streams. The first is an uncompressed buffer, the second is a compressor to a compressed buffer. If checkThreshold is reached, the buffer sizes are compared, and the rest of the data is forwarded only to the final output stream.
type PipeCounter ¶
type PipeCounter struct {
W io.WriteCloser
R io.ReadCloser
PT ProgressFunc
Count int64
}
PipeCounter is a PipeWriter that tracks the number of bytes passing through the stream. This can be handy for monitoring sizes when other pipe stream segments modify the size of the data (eg due to compression).
func NewPipeCounter ¶
func NewPipeCounter() *PipeCounter
NewPipeCounter returns a fresh PipeCounter ready for connection.
func (*PipeCounter) AddProgressTracker ¶
func (pc *PipeCounter) AddProgressTracker(pt ProgressFunc) PipeReader
AddProgressTracker for PipeCounter adds progress tracking functionality to the pipeline.
func (*PipeCounter) Close ¶
func (pc *PipeCounter) Close() error
Close for PipeCounter forwards the close action to either the read or write streams if they are defined.
func (*PipeCounter) ConnectIn ¶
func (pc *PipeCounter) ConnectIn(r io.ReadCloser) PipeReader
ConnectIn for PipeCounter assigns the readcloser to the internal read closer.
func (*PipeCounter) ConnectOut ¶
func (pc *PipeCounter) ConnectOut(w io.WriteCloser) PipeWriter
ConnectOut for PipeCounter assigns the writecloser to the internal write closer.
type PipeCreateFile ¶
PipeCreateFile provides an output termination pipe segment that creates (or overwrites) a file for writing, and closes it when the pipeline is closed.
func (*PipeCreateFile) Close ¶
func (p *PipeCreateFile) Close() error
Close for PipeOut forwards a close operation.
func (*PipeCreateFile) ConnectOut ¶
func (p *PipeCreateFile) ConnectOut(w io.WriteCloser) PipeWriter
ConnectOut for PipeOut assigns the writecloser to the internal write closer.
type PipeDigest ¶
type PipeDigest struct {
W io.WriteCloser
R io.ReadCloser
Digester hash.Hash
}
PipeDigest performs a Sha256 digest on the passing stream. This is done in quasi-parallel, writing unbuffered stream to the digester then to the connected writer Warning: retrieving the digest hash while a write is in progress can cause a panic. It is best to ensure the pipeline is closed (by calling Close() on the 'entrance' segment) before checking the digest.
func NewPipeDigest ¶
func NewPipeDigest() *PipeDigest
NewPipeDigest creates a new pipewriter segment that performs a digest of the passing stream, ready for connection.
func (*PipeDigest) Close ¶
func (pd *PipeDigest) Close() error
Close for PipeDigest forwards the close action to either the read or write streams if they are defined.
func (*PipeDigest) ConnectIn ¶
func (pd *PipeDigest) ConnectIn(r io.ReadCloser) PipeReader
ConnectIn for PipeDigest assigns the readcloser to the internal read closer.
func (*PipeDigest) ConnectOut ¶
func (pd *PipeDigest) ConnectOut(w io.WriteCloser) PipeWriter
ConnectOut for PipeDigest assigns the writecloser to the internal write closer.
func (*PipeDigest) GetDigest ¶
func (pd *PipeDigest) GetDigest() digest.Digest
GetDigest returns the current digest stored in the digest stream writer.
type PipeGzDec ¶
type PipeGzDec struct {
R io.ReadCloser
}
PipeGzDec implements a PipeReader that performs gz encoding on the pipe stream.
func NewPipeGzDec ¶
func NewPipeGzDec() *PipeGzDec
NewPipeGzDec creates a PipeWriter that performs gz decompression on the passing stream.
func (*PipeGzDec) ConnectIn ¶
func (z *PipeGzDec) ConnectIn(r io.ReadCloser) PipeReader
ConnectIn for PipeGzDec wraps the incoming writecloser with a zstd encoder writer.
type PipeGzEnc ¶
type PipeGzEnc struct {
W io.WriteCloser
}
PipeGzEnc implements a PipeWriter that performs zstd encoding on the pipe stream.
func NewPipeGzEnc ¶
func NewPipeGzEnc() *PipeGzEnc
NewPipeGzEnc creates a PipeWriter that performs zstd compression on the passing stream.
func (*PipeGzEnc) ConnectOut ¶
func (z *PipeGzEnc) ConnectOut(w io.WriteCloser) PipeWriter
ConnectOut for PipeGzEnc wraps the incoming writecloser with a zstd encoder writer.
type PipeIn ¶
type PipeIn struct {
R io.ReadCloser
}
PipeIn is a PipeReader wrapping around a standard read closer, forwarding all reads to the associated read closer. This essentially provides a basic no-op forwarding implementation.
func (*PipeIn) ConnectIn ¶
func (p *PipeIn) ConnectIn(r io.ReadCloser) PipeReader
ConnectIn for PipeIn assigns the readcloser to the internal read closer.
type PipeOut ¶
type PipeOut struct {
W io.WriteCloser
}
PipeOut is a PipeWriter wrapping around a standard write closer, forwarding all writes to the associated write closer. This essentially provides a basic no-op forwarding implementation.
func (*PipeOut) ConnectOut ¶
func (p *PipeOut) ConnectOut(w io.WriteCloser) PipeWriter
ConnectOut for PipeOut assigns the writecloser to the internal write closer.
type PipeReader ¶
type PipeReader interface {
io.Reader
io.Closer
// ConnectIn wraps a ReadCloser with additional functionality that can occur during read or close with the
// expectation that the data is read from the up stream wrapped reader, and close operations are forwarded
ConnectIn(r io.ReadCloser) PipeReader
}
PipeReader is an interface that provides the necessary functionality for operating with input pipe streams. This provides a io.ReadCloser compatible interface, and thus can be used as the source for any stream read operation. In addition, the input can be connected to other read closers (commonly, other PipeReader implementations), to daisy chain a collection of read handlers.
type PipeTerm ¶
type PipeTerm struct{}
PipeTerm is a PipeWriter that performs no operation, useful for null-terminating a pipe line.
func (*PipeTerm) ConnectOut ¶
func (pt *PipeTerm) ConnectOut(io.WriteCloser) PipeWriter
ConnectOut for PipeTerm ignores the provided output stream and returns an empty PipeWriter.
type PipeWriter ¶
type PipeWriter interface {
io.Writer
io.Closer
// ConnectOut wraps a WriteCloser with additional functionality that can occur during write or close with the
// expectation that the data is forwarded down stream to the wrapped writer, as well as close operations
ConnectOut(w io.WriteCloser) PipeWriter
}
PipeWriter is an interface that provides the necessary functionality for operating with output pipe streams. This provides a io.WriteCloser compatible interface, and thus can be used as the destination for any stream write operation. In addition, the output can be connected to other write closers (commonly, other PipeWriter implementations), to daisy chain a collection of write handlers.
func NewPipeFileCreator ¶
func NewPipeFileCreator(path string) (PipeWriter, error)
NewPipeFileCreator creates a PipeCreateFile pipe segment. Internally, a new file with the specified path is created or overwritten, ready for writing.
type PipeZstdDec ¶
type PipeZstdDec struct {
// contains filtered or unexported fields
}
PipeZstdDec implements a PipeReader that performs zstd encoding on the pipe stream.
func NewPipeZstdDec ¶
func NewPipeZstdDec() *PipeZstdDec
NewPipeZstdDec creates a PipeWriter that performs zstd decompression on the passing stream.
func (*PipeZstdDec) Close ¶
func (z *PipeZstdDec) Close() error
Close for PipeZstdDec closes the encoder.
func (*PipeZstdDec) ConnectIn ¶
func (z *PipeZstdDec) ConnectIn(r io.ReadCloser) PipeReader
ConnectIn for PipeZstdDec wraps the incoming writecloser with a zstd encoder writer.
type PipeZstdEnc ¶
type PipeZstdEnc struct {
// contains filtered or unexported fields
}
PipeZstdEnc implements a PipeWriter that performs zstd encoding on the pipe stream.
func NewPipeZstdEnc ¶
func NewPipeZstdEnc() *PipeZstdEnc
NewPipeZstdEnc creates a PipeWriter that performs zstd compression on the passing stream.
func NewPipeZstdEncLevel ¶
func NewPipeZstdEncLevel(lv zstd.EncoderLevel) *PipeZstdEnc
NewPipeZstdEncLevel creates a PipeWriter that performs zstd compression on the passing stream at the specified level.
func (*PipeZstdEnc) Close ¶
func (z *PipeZstdEnc) Close() error
Close for PipeZstdEnc closes the encoder.
func (*PipeZstdEnc) ConnectOut ¶
func (z *PipeZstdEnc) ConnectOut(w io.WriteCloser) PipeWriter
ConnectOut for PipeZstdEnc wraps the incoming writecloser with a zstd encoder writer.
func (*PipeZstdEnc) ConnectOutWithLevel ¶
func (z *PipeZstdEnc) ConnectOutWithLevel(w io.WriteCloser, lv zstd.EncoderLevel) PipeWriter
ConnectOutWithLevel for PipeZstdEnc wraps the incoming writecloser with a customized zstd encoder writer.
type ProgressFunc ¶
type ProgressFunc func(complete int64)
ProgressFunc is callback function used to update the number of bytes complete.
type TarArchiver ¶
type TarArchiver struct {
// contains filtered or unexported fields
}
TarArchiver archives a directory to a stream.
func NewArchiver ¶
func NewArchiver(ostream io.Writer, options ...TarArchiverOption) (*TarArchiver, error)
NewArchiver initializes a new archive and prepares it for writing.
func (*TarArchiver) Close ¶
func (ar *TarArchiver) Close() error
Close closes the writer if present, and clears any existing stream objects.
func (*TarArchiver) WriteEntry ¶
WriteEntry adds file or directory data to the archive. finfo is expected to be the FileInfo for the actual file or directory (not a symlink to a file or directory).
type TarArchiverOption ¶
type TarArchiverOption func(ar *TarArchiver) error
TarArchiverOption is used for options for the archival process.
func WithExecutableFiles ¶
func WithExecutableFiles() TarArchiverOption
WithExecutableFiles perserves executable file within the archive (the default is to remove executable permissions on all files).
func WithPortablePaths ¶
func WithPortablePaths() TarArchiverOption
WithPortablePaths ensures that all paths within the archive are portable paths (for cross platform compatibility) We do not want to have file within archives having bad file names that do not work on Windows.
type TarExtractor ¶
type TarExtractor struct {
// OverwriteExisting files when extracting
OverwriteExisting bool
// MakeParentDirectories will make all parents instead of just what should be necessary (set to true for backwards compatibility)
MakeParentDirectories bool
// contains filtered or unexported fields
}
TarExtractor is used for reading a tar archive from a stream and extracting to disk.
func (*TarExtractor) Open ¶
func (ar *TarExtractor) Open(in io.Reader) error
Open opens an archive stream for reading.
func (*TarExtractor) Read ¶
func (ar *TarExtractor) Read() (*TarFileData, error)
Read reads the next file from t, which must have already been opened for reading. If there are no more files, the error.Is io.EOF. The File must be closed when finished reading from it.
func (*TarExtractor) UnarchiveRead ¶
func (ar *TarExtractor) UnarchiveRead(sourceReader io.Reader, destPath string) error
UnarchiveRead implements tar unarchiving from an io.reader stream to a destination path.
type TarFileData ¶
type TarFileData struct {
fs.FileInfo
// The original header info; depends on
// type of archive -- could be nil, too.
Header *tar.Header
// Allow the file contents to be read (and closed)
io.ReadCloser
}
TarFileData provides methods for accessing information about or contents of a file within an archive.