archive

package
v1.16.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 15, 2025 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package archive implements methods for working with archives, including archiving and compressing tar archives using tar and zstd libraries. Additionally, a stream pipeline is defined that enables archival, compression, and digesting operations to occur in a modular fashion with one pass through a file.

Index

Constants

This section is empty.

Variables

View Source
var ErrNonPortablePath = errors.New("non-portable path")

ErrNonPortablePath is used to denote a filename/path that contains invalid characters.

Functions

func AssignCompressionLevel

func AssignCompressionLevel(userSpecifiedLevel string) zstd.EncoderLevel

AssignCompressionLevel Helper function to assign determine custom encoding level.

func EncoderWithCustomLevel

func EncoderWithCustomLevel(lv zstd.EncoderLevel) *zstd.Encoder

EncoderWithCustomLevel returns an Encoder at the specified compression level.

func ExtractTar

func ExtractTar(ctx context.Context, rc io.ReadCloser, destPath string) error

ExtractTar unarchives a tar file. Existing files are overwritten.

func ExtractTarCompat

func ExtractTarCompat(ctx context.Context, rc io.ReadCloser, destPath string) error

ExtractTarCompat unarchives a tar file. Existing files are overwritten. Parent directories are created as needed.

func ExtractTarZstd

func ExtractTarZstd(ctx context.Context, rc io.ReadCloser, destPath string) error

ExtractTarZstd decompresses and unarchives a tar+zst file. Existing files are overwritten.

func ExtractTarZstdCompat

func ExtractTarZstdCompat(ctx context.Context, rc io.ReadCloser, destPath string) error

ExtractTarZstdCompat decompresses and unarchives a tar+zst file. Existing files are overwritten. Parent directories are created as needed.

func ExtractZstd

func ExtractZstd(ctx context.Context, rc io.ReadCloser, destPath string) error

ExtractZstd decompresses a .zst file. Existing files are overwritten.

func TarToStream

func TarToStream(ctx context.Context, fsys fs.FS, ostream io.Writer, options ...TarArchiverOption) error

TarToStream archives a list of files and writes the archive to the provided stream.

Types

type BufWriteCloser

type BufWriteCloser struct {
	*bytes.Buffer
}

BufWriteCloser is a wrapper around bytes.buffer that provides a no-op close for io.WriteCloser compatibility.

func (*BufWriteCloser) Close

func (bwc *BufWriteCloser) Close() error

Close for BufWriteCloser doesn't need to do anything, just for interface compat.

type FlushWriter

type FlushWriter interface {
	io.Writer
	io.Closer
	Flush() error
}

FlushWriter combines closer, writer, and Flush, used for tracking compressors that must flush data before checking the output size.

type PipeCompressThreshold

type PipeCompressThreshold struct {
	W io.WriteCloser // final output writer

	FinalRatio  float32 // if nonzero, this is the resulting ratio  (uncompressed/compressed) for reporting purposes
	DidCompress bool    // true if the compression ratio check passed and the data was compressed.
	// contains filtered or unexported fields
}

PipeCompressThreshold performs compression on input and forwards compressed data to the output, as long as the compression ratio meets a specified threshold. If the compression threshold is NOT met, the data is instead piped directly to the output, uncompressed. To perform this, the first N bytes of the input are buffered, then compressed into a second buffer. When the data size reaches N, the size of the two buffers are compared to determine the ratio. If the ratio meets desired threshold, the compressed buffer is forwarded to the output pipe, and incoming data is no longer buffered. Conversely, if the ratio is not met, the uncompressed buffer is forwarded to the output, and the compression process is bypassed for the remainder of the data.

func NewPipeCompressThreshold

func NewPipeCompressThreshold(lvl zstd.EncoderLevel, checkSize int, useGz bool) *PipeCompressThreshold

NewPipeCompressThreshold creates a pipe writer that will buffer the initial checkSize bytes, keeping two versions, a compressed and uncompressed one. When the checkSize is met, the compressibility of the data is examined. If the data is not compressible, then the uncompressed data is output. Further writes after the check will no longer buffer data, and will instead pipe data directly to the output (or compressor). lvl indicates a zstd compression level to use. Currently, levels are only supported for zstd, and lvl is ignored for Gzip. useGz should be set to true to use a gzip encoder versus the zstd one.

func (*PipeCompressThreshold) Close

func (pct *PipeCompressThreshold) Close() error

Close for PipeCompressThreshold will simply forward a close operation down the pipe if the compression check threshold has already been met. If that threshold has not been met, however, we have to flush the compressed or uncompressed buffer to the output based on compression ratio.

func (*PipeCompressThreshold) ConnectOut

func (pct *PipeCompressThreshold) ConnectOut(w io.WriteCloser) PipeWriter

ConnectOut establishes the temporary buffers, and sets the final output for when a compression decision has been reached. Until the threshold is reached, data is multiwritten to two temporary buffered outputs. Due to this buffering, output from PipeCompressThreshold is delayed (the delay goes away once the threshold is met). Ultimately, the data will be either compressed or uncompressed depending on the threshold calculation.

func (*PipeCompressThreshold) IsCompressible

func (pct *PipeCompressThreshold) IsCompressible() bool

IsCompressible returns true if the ratio compressed/uncompressed is less than a threshold. This is safe to call after the write process is complete to determine if the initial data was compressible.

func (*PipeCompressThreshold) Write

func (pct *PipeCompressThreshold) Write(p []byte) (n int, err error)

Write for PipeCompressThreshold writes incoming data to two output streams. The first is an uncompressed buffer, the second is a compressor to a compressed buffer. If checkThreshold is reached, the buffer sizes are compared, and the rest of the data is forwarded only to the final output stream.

type PipeCounter

type PipeCounter struct {
	W     io.WriteCloser
	R     io.ReadCloser
	PT    ProgressFunc
	Count int64
}

PipeCounter is a PipeWriter that tracks the number of bytes passing through the stream. This can be handy for monitoring sizes when other pipe stream segments modify the size of the data (eg due to compression).

func NewPipeCounter

func NewPipeCounter() *PipeCounter

NewPipeCounter returns a fresh PipeCounter ready for connection.

func (*PipeCounter) AddProgressTracker

func (pc *PipeCounter) AddProgressTracker(pt ProgressFunc) PipeReader

AddProgressTracker for PipeCounter adds progress tracking functionality to the pipeline.

func (*PipeCounter) Close

func (pc *PipeCounter) Close() error

Close for PipeCounter forwards the close action to either the read or write streams if they are defined.

func (*PipeCounter) ConnectIn

func (pc *PipeCounter) ConnectIn(r io.ReadCloser) PipeReader

ConnectIn for PipeCounter assigns the readcloser to the internal read closer.

func (*PipeCounter) ConnectOut

func (pc *PipeCounter) ConnectOut(w io.WriteCloser) PipeWriter

ConnectOut for PipeCounter assigns the writecloser to the internal write closer.

func (*PipeCounter) Read

func (pc *PipeCounter) Read(p []byte) (n int, err error)

Read for PipeCounter increments the counter, and forwards the output to the internal reader.

func (*PipeCounter) Write

func (pc *PipeCounter) Write(p []byte) (n int, err error)

Write for PipeCounter increments the counter, and forwards the output to the internal writer.

type PipeCreateFile

type PipeCreateFile struct {
	F *os.File
}

PipeCreateFile provides an output termination pipe segment that creates (or overwrites) a file for writing, and closes it when the pipeline is closed.

func (*PipeCreateFile) Close

func (p *PipeCreateFile) Close() error

Close for PipeOut forwards a close operation.

func (*PipeCreateFile) ConnectOut

func (p *PipeCreateFile) ConnectOut(w io.WriteCloser) PipeWriter

ConnectOut for PipeOut assigns the writecloser to the internal write closer.

func (*PipeCreateFile) Write

func (p *PipeCreateFile) Write(b []byte) (n int, err error)

Write for PipeOut forwards writes to the internal write closer.

type PipeDigest

type PipeDigest struct {
	W        io.WriteCloser
	R        io.ReadCloser
	Digester hash.Hash
}

PipeDigest performs a Sha256 digest on the passing stream. This is done in quasi-parallel, writing unbuffered stream to the digester then to the connected writer Warning: retrieving the digest hash while a write is in progress can cause a panic. It is best to ensure the pipeline is closed (by calling Close() on the 'entrance' segment) before checking the digest.

func NewPipeDigest

func NewPipeDigest() *PipeDigest

NewPipeDigest creates a new pipewriter segment that performs a digest of the passing stream, ready for connection.

func (*PipeDigest) Close

func (pd *PipeDigest) Close() error

Close for PipeDigest forwards the close action to either the read or write streams if they are defined.

func (*PipeDigest) ConnectIn

func (pd *PipeDigest) ConnectIn(r io.ReadCloser) PipeReader

ConnectIn for PipeDigest assigns the readcloser to the internal read closer.

func (*PipeDigest) ConnectOut

func (pd *PipeDigest) ConnectOut(w io.WriteCloser) PipeWriter

ConnectOut for PipeDigest assigns the writecloser to the internal write closer.

func (*PipeDigest) GetDigest

func (pd *PipeDigest) GetDigest() digest.Digest

GetDigest returns the current digest stored in the digest stream writer.

func (*PipeDigest) Read

func (pd *PipeDigest) Read(p []byte) (n int, err error)

Read for PipeDigest calculates the digest by writing to the digester each chunk of bytes is read from the reader, and then written to the digester for the calculation.

func (*PipeDigest) Write

func (pd *PipeDigest) Write(p []byte) (n int, err error)

Write for PipeDigest calculates the digest by writing to the digester each chunk of bytes is written to the digester and then to the connected writer.

type PipeGzDec

type PipeGzDec struct {
	R io.ReadCloser
}

PipeGzDec implements a PipeReader that performs gz encoding on the pipe stream.

func NewPipeGzDec

func NewPipeGzDec() *PipeGzDec

NewPipeGzDec creates a PipeWriter that performs gz decompression on the passing stream.

func (*PipeGzDec) Close

func (z *PipeGzDec) Close() error

Close for PipeGzDec forwards a close operation.

func (*PipeGzDec) ConnectIn

func (z *PipeGzDec) ConnectIn(r io.ReadCloser) PipeReader

ConnectIn for PipeGzDec wraps the incoming writecloser with a zstd encoder writer.

func (*PipeGzDec) Read

func (z *PipeGzDec) Read(d []byte) (n int, err error)

Read for PipeGzDec forwards reads to the internal read closer.

type PipeGzEnc

type PipeGzEnc struct {
	W io.WriteCloser
}

PipeGzEnc implements a PipeWriter that performs zstd encoding on the pipe stream.

func NewPipeGzEnc

func NewPipeGzEnc() *PipeGzEnc

NewPipeGzEnc creates a PipeWriter that performs zstd compression on the passing stream.

func (*PipeGzEnc) Close

func (z *PipeGzEnc) Close() error

Close for PipeGzEnc forwards a close operation.

func (*PipeGzEnc) ConnectOut

func (z *PipeGzEnc) ConnectOut(w io.WriteCloser) PipeWriter

ConnectOut for PipeGzEnc wraps the incoming writecloser with a zstd encoder writer.

func (*PipeGzEnc) Write

func (z *PipeGzEnc) Write(b []byte) (n int, err error)

Write for PipeGzEnc forwards writes to the internal write closer.

type PipeIn

type PipeIn struct {
	R io.ReadCloser
}

PipeIn is a PipeReader wrapping around a standard read closer, forwarding all reads to the associated read closer. This essentially provides a basic no-op forwarding implementation.

func (*PipeIn) Close

func (p *PipeIn) Close() error

Close for PipeIn forwards a close operation.

func (*PipeIn) ConnectIn

func (p *PipeIn) ConnectIn(r io.ReadCloser) PipeReader

ConnectIn for PipeIn assigns the readcloser to the internal read closer.

func (*PipeIn) Read

func (p *PipeIn) Read(d []byte) (n int, err error)

Read for PipeIn forwards reads to the internal read closer.

type PipeOut

type PipeOut struct {
	W io.WriteCloser
}

PipeOut is a PipeWriter wrapping around a standard write closer, forwarding all writes to the associated write closer. This essentially provides a basic no-op forwarding implementation.

func (*PipeOut) Close

func (p *PipeOut) Close() error

Close for PipeOut forwards a close operation.

func (*PipeOut) ConnectOut

func (p *PipeOut) ConnectOut(w io.WriteCloser) PipeWriter

ConnectOut for PipeOut assigns the writecloser to the internal write closer.

func (*PipeOut) Write

func (p *PipeOut) Write(b []byte) (n int, err error)

Write for PipeOut forwards writes to the internal write closer.

type PipeReader

type PipeReader interface {
	io.Reader
	io.Closer
	// ConnectIn wraps a ReadCloser with additional functionality that can occur during read or close with the
	// expectation that the data is read from the up stream wrapped reader, and close operations are forwarded
	ConnectIn(r io.ReadCloser) PipeReader
}

PipeReader is an interface that provides the necessary functionality for operating with input pipe streams. This provides a io.ReadCloser compatible interface, and thus can be used as the source for any stream read operation. In addition, the input can be connected to other read closers (commonly, other PipeReader implementations), to daisy chain a collection of read handlers.

type PipeTerm

type PipeTerm struct{}

PipeTerm is a PipeWriter that performs no operation, useful for null-terminating a pipe line.

func (*PipeTerm) Close

func (pt *PipeTerm) Close() error

Close for PipeTerm does nothing.

func (*PipeTerm) ConnectOut

func (pt *PipeTerm) ConnectOut(io.WriteCloser) PipeWriter

ConnectOut for PipeTerm ignores the provided output stream and returns an empty PipeWriter.

func (*PipeTerm) Write

func (pt *PipeTerm) Write(b []byte) (n int, err error)

Write for PipeTerm does nothing.

type PipeWriter

type PipeWriter interface {
	io.Writer
	io.Closer
	// ConnectOut wraps a WriteCloser with additional functionality that can occur during write or close with the
	// expectation that the data is forwarded down stream to the wrapped writer, as well as close operations
	ConnectOut(w io.WriteCloser) PipeWriter
}

PipeWriter is an interface that provides the necessary functionality for operating with output pipe streams. This provides a io.WriteCloser compatible interface, and thus can be used as the destination for any stream write operation. In addition, the output can be connected to other write closers (commonly, other PipeWriter implementations), to daisy chain a collection of write handlers.

func NewPipeFileCreator

func NewPipeFileCreator(path string) (PipeWriter, error)

NewPipeFileCreator creates a PipeCreateFile pipe segment. Internally, a new file with the specified path is created or overwritten, ready for writing.

type PipeZstdDec

type PipeZstdDec struct {
	// contains filtered or unexported fields
}

PipeZstdDec implements a PipeReader that performs zstd encoding on the pipe stream.

func NewPipeZstdDec

func NewPipeZstdDec() *PipeZstdDec

NewPipeZstdDec creates a PipeWriter that performs zstd decompression on the passing stream.

func (*PipeZstdDec) Close

func (z *PipeZstdDec) Close() error

Close for PipeZstdDec closes the encoder.

func (*PipeZstdDec) ConnectIn

func (z *PipeZstdDec) ConnectIn(r io.ReadCloser) PipeReader

ConnectIn for PipeZstdDec wraps the incoming writecloser with a zstd encoder writer.

func (*PipeZstdDec) Read

func (z *PipeZstdDec) Read(p []byte) (n int, err error)

Read for PipeZstdDec reads data from the zstd decoder (which in turn reads encoded data from the original input reader).

type PipeZstdEnc

type PipeZstdEnc struct {
	// contains filtered or unexported fields
}

PipeZstdEnc implements a PipeWriter that performs zstd encoding on the pipe stream.

func NewPipeZstdEnc

func NewPipeZstdEnc() *PipeZstdEnc

NewPipeZstdEnc creates a PipeWriter that performs zstd compression on the passing stream.

func NewPipeZstdEncLevel

func NewPipeZstdEncLevel(lv zstd.EncoderLevel) *PipeZstdEnc

NewPipeZstdEncLevel creates a PipeWriter that performs zstd compression on the passing stream at the specified level.

func (*PipeZstdEnc) Close

func (z *PipeZstdEnc) Close() error

Close for PipeZstdEnc closes the encoder.

func (*PipeZstdEnc) ConnectOut

func (z *PipeZstdEnc) ConnectOut(w io.WriteCloser) PipeWriter

ConnectOut for PipeZstdEnc wraps the incoming writecloser with a zstd encoder writer.

func (*PipeZstdEnc) ConnectOutWithLevel

func (z *PipeZstdEnc) ConnectOutWithLevel(w io.WriteCloser, lv zstd.EncoderLevel) PipeWriter

ConnectOutWithLevel for PipeZstdEnc wraps the incoming writecloser with a customized zstd encoder writer.

func (*PipeZstdEnc) Write

func (z *PipeZstdEnc) Write(p []byte) (n int, err error)

Write for PipeZstdEnc writes data to the zstd encoder (which in turn writes encoded data to the original output writer).

type ProgressFunc

type ProgressFunc func(complete int64)

ProgressFunc is callback function used to update the number of bytes complete.

type TarArchiver

type TarArchiver struct {
	// contains filtered or unexported fields
}

TarArchiver archives a directory to a stream.

func NewArchiver

func NewArchiver(ostream io.Writer, options ...TarArchiverOption) (*TarArchiver, error)

NewArchiver initializes a new archive and prepares it for writing.

func (*TarArchiver) Close

func (ar *TarArchiver) Close() error

Close closes the writer if present, and clears any existing stream objects.

func (*TarArchiver) WriteEntry

func (ar *TarArchiver) WriteEntry(fsys fs.FS, path string, finfo fs.FileInfo) error

WriteEntry adds file or directory data to the archive. finfo is expected to be the FileInfo for the actual file or directory (not a symlink to a file or directory).

type TarArchiverOption

type TarArchiverOption func(ar *TarArchiver) error

TarArchiverOption is used for options for the archival process.

func WithExecutableFiles

func WithExecutableFiles() TarArchiverOption

WithExecutableFiles perserves executable file within the archive (the default is to remove executable permissions on all files).

func WithPortablePaths

func WithPortablePaths() TarArchiverOption

WithPortablePaths ensures that all paths within the archive are portable paths (for cross platform compatibility) We do not want to have file within archives having bad file names that do not work on Windows.

type TarExtractor

type TarExtractor struct {

	// OverwriteExisting files when extracting
	OverwriteExisting bool

	// MakeParentDirectories will make all parents instead of just what should be necessary (set to true for backwards compatibility)
	MakeParentDirectories bool
	// contains filtered or unexported fields
}

TarExtractor is used for reading a tar archive from a stream and extracting to disk.

func (*TarExtractor) Open

func (ar *TarExtractor) Open(in io.Reader) error

Open opens an archive stream for reading.

func (*TarExtractor) Read

func (ar *TarExtractor) Read() (*TarFileData, error)

Read reads the next file from t, which must have already been opened for reading. If there are no more files, the error.Is io.EOF. The File must be closed when finished reading from it.

func (*TarExtractor) UnarchiveRead

func (ar *TarExtractor) UnarchiveRead(sourceReader io.Reader, destPath string) error

UnarchiveRead implements tar unarchiving from an io.reader stream to a destination path.

type TarFileData

type TarFileData struct {
	fs.FileInfo

	// The original header info; depends on
	// type of archive -- could be nil, too.
	Header *tar.Header

	// Allow the file contents to be read (and closed)
	io.ReadCloser
}

TarFileData provides methods for accessing information about or contents of a file within an archive.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL