40 | interfaces and tools in io package (I)

In previous articles, we mainly discussed three data types: strings.Builder, strings.Reader and bytes.Buffer.

Knowledge review

Remember? I also asked you "what interfaces do they implement". Before we continue to explain the interfaces and tools in the io package, let me answer this question first.

strings.Builder type is mainly used to build strings. Its pointer type implements io.Writer, io.ByteWriter and fmt.Stringer interfaces. In addition, it actually implements a package level private interface io.stringwriter of IO package (it will be renamed io.stringwriter since Go 1.12).

strings.Reader type is mainly used to read strings. Its pointer type implements many interfaces, including:

• io.Reader；
• io.ReaderAt；
• io.ByteReader；
• io.RuneReader；
• io.Seeker；
• io.ByteScanner；
• io.RuneScanner；
• io.WriterTo；

There are 8 interfaces in total, all of which are interfaces in the io package.

Among them, io.ByteScanner is the extension interface of io.ByteReader, and io.RuneScanner is the extension interface of io.RuneReader.

bytes.Buffer is a data type integrating read and write functions. It is very suitable as a buffer for byte sequence. Its pointer type implements more interfaces.

More specifically, the pointer type implements the following interfaces related to reading.

• io.Reader；
• io.ByteReader；
• io.RuneReader；
• io.ByteScanner；
• io.RuneScanner；
• io.WriterTo；

There are six. These are the write related interfaces implemented.

• io.Writer；
• io.ByteWriter；
• io.stringWriter；
• io.ReaderFrom；

4 in total. In addition, it also implements the export related interface fmt.Stringer.

Leading content: benefits and advantages of interface in io package

So what is the motivation (or purpose) for these types to implement so many interfaces?

In short, this is to improve the interoperability between different program entities. Far from that, let's take some functions in the io package as examples.

In the io package, there are several functions for copying data, which are:

• io.Copy；
• io.CopyBuffer；
• io.CopyN.

Although these functions are slightly different in function, they will first accept two parameters: the parameter dst representing the data destination and io.Writer type, and the parameter src representing the data source and io.Reader type. These functions basically copy data from src to dst.

No matter what type the first parameter value we give them, as long as this type implements the io.Writer interface.

Similarly, no matter what the actual type of the second parameter value we pass them, as long as the type implements the io.Reader interface.

Once we meet these two conditions, these functions can be executed almost normally. Of course, the function will also check the validity of the necessary parameter values. If the check fails, its execution will not end successfully.

Here is a sample code:

src := strings.NewReader(
 "CopyN copies n bytes (or until an error) from src to dst. " +
  "It returns the number of bytes copied and " +
  "the earliest error encountered while copying.")
dst := new(strings.Builder)
written, err := io.CopyN(dst, src, 58)
if err != nil {
 fmt.Printf("error: %v\n", err)
} else {
 fmt.Printf("Written(%d): %q\n", written, dst.String())
}

I first created a string reader using strings.NewReader and assigned it to the variable src. Then I created a string builder and assigned it to the variable dst.

Later, when I called the io.CopyN function, I passed in the values of these two variables, and set the third parameter value to 58. That is, I want to copy the first 58 bytes from src to dst.

Although the types of variables src and dst are strings.Reader and strings.Builder respectively, when they are passed to the io.CopyN function, they have been wrapped into the values of io.Reader and io.Writer respectively. io.CopyN functions don't care what their actual types are.

For the purpose of optimization, the code in the io.CopyN function repackages the parameter values, detects whether these parameter values implement other interfaces, and even explores whether the actual type of a parameter value packaged is a special type.

However, on the whole, these codes are oriented to the interface in the parameter declaration. The author of io.CopyN function has greatly expanded its scope of application and application scenarios through interface oriented programming.

From another perspective, it is precisely because the strings.Reader type and the strings.Builder type implement many interfaces that their values can be used in a broader scenario.

In other words, there are significantly more functions and data types that can manipulate them in various libraries of Go language.

This is what I want to tell you. The data types in strings package and bytes package get the greatest benefit after implementing several interfaces.

In other words, this is the biggest advantage of interface oriented programming. The practice of these data types and functions is also very worthy of our imitation in the process of programming.

As you can see, most of the types described above implement the interfaces in the io code package. In fact, the interface in the io package plays an important role in the standard library of Go language and many third-party libraries. They are very basic and important.

Take io.Reader and io.Writer, the two core interfaces, for example. They are extension objects and design sources of many interfaces. At the same time, according to the statistics from the standard library of Go language, there are hundreds of data types (each) and more than 400 codes referencing them.

Many data types implement the io.Reader interface because they provide the ability to read data from somewhere. Similarly, many data types that can write data somewhere will also implement the io.Writer interface.

In fact, the original design intention of many types is to implement one or some extension interfaces of the two core interfaces to provide richer functions than simple byte sequence reading or writing, just like the data types in the strings package and bytes package mentioned above.

In Go language, the extension of interface is realized by embedding between interface types, which is often called interface combination.

When I talked about interfaces, I also mentioned that Go language advocates the use of small interfaces and interface combination to expand program behavior and increase program flexibility. io code package can be used as such a benchmark, which can become a reference standard when we use this technique.

package main

import (
	"bytes"
	"fmt"
	"io"
	"strings"
)

func main() {
	// Example 1.
	builder := new(strings.Builder)
	_ = interface{}(builder).(io.Writer)
	_ = interface{}(builder).(io.ByteWriter)
	_ = interface{}(builder).(fmt.Stringer)

	// Example 2.
	reader := strings.NewReader("")
	_ = interface{}(reader).(io.Reader)
	_ = interface{}(reader).(io.ReaderAt)
	_ = interface{}(reader).(io.ByteReader)
	_ = interface{}(reader).(io.RuneReader)
	_ = interface{}(reader).(io.Seeker)
	_ = interface{}(reader).(io.ByteScanner)
	_ = interface{}(reader).(io.RuneScanner)
	_ = interface{}(reader).(io.WriterTo)

	// Example 3.
	buffer := bytes.NewBuffer([]byte{})
	_ = interface{}(buffer).(io.Reader)
	_ = interface{}(buffer).(io.ByteReader)
	_ = interface{}(buffer).(io.RuneReader)
	_ = interface{}(buffer).(io.ByteScanner)
	_ = interface{}(buffer).(io.RuneScanner)
	_ = interface{}(buffer).(io.WriterTo)

	_ = interface{}(buffer).(io.Writer)
	_ = interface{}(buffer).(io.ByteWriter)
	_ = interface{}(buffer).(io.ReaderFrom)

	_ = interface{}(buffer).(fmt.Stringer)

	// Example 4.
	src := strings.NewReader(
		"CopyN copies n bytes (or until an error) from src to dst. " +
			"It returns the number of bytes copied and " +
			"the earliest error encountered while copying.")
	dst := new(strings.Builder)
	written, err := io.CopyN(dst, src, 58)
	if err != nil {
		fmt.Printf("error: %v\n", err)
	} else {
		fmt.Printf("Written(%d): %q\n", written, dst.String())
	}
}

Next, I will take the io.Reader interface as the object and raise a problem related to interface extension and implementation. If you have studied the core interface and related data types, it is not difficult to answer this question.

Our question today is: what are the extension interfaces and implementation types of io.Reader in the io package? What are their functions?

The typical answer to this question is this. In the io package, io.Reader has the following extension interfaces.

1. io.ReadWriter: this interface is not only the extension interface of io.Reader, but also the extension interface of io.Writer. In other words, the interface defines a set of behaviors, including and only the basic byte sequence Read method and byte sequence Write method.

2. io.ReadCloser: in addition to the basic byte sequence reading method, this interface also has a basic Close method. The latter is generally used to Close the path of data reading and writing. This interface is actually a combination of io.Reader interface and io.Closer interface.

3. io.ReadWriteCloser: obviously, this interface is a combination of io.Reader, io.Writer and io.Closer.

4. io.ReadSeeker: the feature of this interface is that it has a basic method seek for finding read-write locations. More specifically, the method can find a new read-write position based on the start position, end position, or current read-write position of the data according to a given offset. This new read / write position is used to indicate the starting index for the next read or write. Seek is the only method owned by the io.Seeker interface.

5. io.ReadWriteSeeker: obviously, this interface is another three in one extension interface. It is a combination of io.Reader, io.Writer and io.Seeker.

Let's talk about the implementation types of the io.Reader interface in the io package. They include the following items.

1. * io.LimitedReader: the basic type of this type will wrap the value of io.Reader type and provide an additional function of restricted reading. The so-called restricted Read means that the total amount of data returned by this type of Read method Read will be limited, no matter how many times the method is called. This limit is indicated by the type of field N in bytes.

2. * io.SectionReader: the basic type of this type can wrap the value of io.ReaderAt type, and its Read method will be limited. It can only Read a part (or a paragraph) of the original data. The start and end positions of this data segment need to be indicated when it is initialized, and cannot be changed later. This type of value behaves somewhat like a slice, and it only exposes the data exposed in its window.

3. * io.TeeReader: this type is a package level private data type and the actual type of the result value of the io.TeeReader function. This function takes two parameters r and W, and the types are io.Reader and io.Writer respectively. The Read method of the result value will write the data in r to w through the byte slice P as the method parameter. It can be said that this value is the data bridge between r and W, and the parameter p is the data carrier on this bridge.

4. * io.multiReader: this type is also a packet level private data type. Similarly, there is a function named MultiReader in the IO package, which can accept several parameter values of io.Reader type and return a result value with the actual type of io.multiReader. When the Read method of this result value is called, it will sequentially Read data from the previous parameter values of io.Reader type. Therefore, we can also call it multi object reader.

5. Io.pipe: this type is a package level private data type, which is much more complex than the above types. It implements not only the io.Reader interface, but also the io.Writer interface. In fact, all pointer methods owned by io.PipeReader and io.PipeWriter types are based on it. These methods only represent a method owned by the IO. Pipe type value. Because the io.pipe function returns the pointer values of these two types and takes them as both ends of the generated synchronous memory pipe, it can be said that the io.pipe type is the core implementation of the synchronous memory pipe provided by the IO package.

6. * io.PipeReader: this type can be regarded as the proxy type of io.pipe type. It proxies some functions of the latter, and implements the io.ReadCloser interface based on the latter. At the same time, it also defines the read end of the synchronous memory pipe.

Note that I ignore the implementation types in the test source file and those that will not be exposed directly in any form.

Problem analysis

The purpose of this question is to assess your familiarity with io packages. This code package is the foundation of all I/O related API s in the Go language standard library, so we must understand each program entity.

However, because the package contains many contents, the problem here takes the io.Reader interface as the entry point. Through the io.Reader interface, we should be able to sort out the type tree based on it and know the functions of each type.

io.Reader is the core interface in the io package and even the whole Go language standard library, so we can involve many extension interfaces and implementation types from it.

In the typical answer to this question, I listed and introduced the relevant data types within the io package for you.

Each of these types deserves your careful understanding, especially those types that implement the io.Reader interface. Their functions are different in detail.

In many cases, we can match them according to actual needs.

For example, the Read function (provided by the Read method) applied on the original data is multi-level packaged (such as restricted reading and multi object reading) to meet the more complex reading requirements.

In the actual interview, as long as the candidate can start from a certain aspect, say the extension interface of io.Reader and its significance, or clarify the three or five implementation types of the interface, it can be regarded as basically correct.

For example, starting from the basic functions of reading, writing and closing, describe them clearly: io.ReadWriter; io.ReadCloser； io.ReadWriteCloser； These interfaces.

• io.ReadWriter；
• io.ReadCloser；
• io.ReadWriteCloser；

These interfaces.

For another example, understand the similarities and differences between io.LimitedReader and io.SectionReader.

For another example, explain how the * io.SectionReader type implements the io.ReadSeeker interface, and so on. However, this is only a qualified threshold. The more comprehensive the candidate answers, the better.

I wrote some code in the sample file demo82.go to show some basic usage of the above types for your reference.

package main

import (
	"fmt"
	"io"
	"strings"
	"sync"
	"time"
)

func main() {
	comment := "Package io provides basic interfaces to I/O primitives. " +
		"Its primary job is to wrap existing implementations of such primitives, " +
		"such as those in package os, " +
		"into shared public interfaces that abstract the functionality, " +
		"plus some other related primitives."

	// Example 1.
	fmt.Println("New a string reader and name it \"reader1\" ...")
	reader1 := strings.NewReader(comment)
	buf1 := make([]byte, 7)
	n, err := reader1.Read(buf1)
	var offset1, index1 int64
	executeIfNoErr(err, func() {
		fmt.Printf("Read(%d): %q\n", n, buf1[:n])
		offset1 = int64(53)
		index1, err = reader1.Seek(offset1, io.SeekCurrent)
	})
	executeIfNoErr(err, func() {
		fmt.Printf("The new index after seeking from current with offset %d: %d\n",
			offset1, index1)
		n, err = reader1.Read(buf1)
	})
	executeIfNoErr(err, func() {
		fmt.Printf("Read(%d): %q\n", n, buf1[:n])
	})
	fmt.Println()

	// Example 2.
	reader1.Reset(comment)
	num1 := int64(7)
	fmt.Printf("New a limited reader with reader1 and number %d ...\n", num1)
	reader2 := io.LimitReader(reader1, 7)
	buf2 := make([]byte, 10)
	for i := 0; i < 3; i++ {
		n, err = reader2.Read(buf2)
		executeIfNoErr(err, func() {
			fmt.Printf("Read(%d): %q\n", n, buf2[:n])
		})
	}
	fmt.Println()

	// Example 3.
	reader1.Reset(comment)
	offset2 := int64(56)
	num2 := int64(72)
	fmt.Printf("New a section reader with reader1, offset %d and number %d ...\n", offset2, num2)
	reader3 := io.NewSectionReader(reader1, offset2, num2)
	buf3 := make([]byte, 20)
	for i := 0; i < 5; i++ {
		n, err = reader3.Read(buf3)
		executeIfNoErr(err, func() {
			fmt.Printf("Read(%d): %q\n", n, buf3[:n])
		})
	}
	fmt.Println()

	// Example 4.
	reader1.Reset(comment)
	writer1 := new(strings.Builder)
	fmt.Println("New a tee reader with reader1 and writer1 ...")
	reader4 := io.TeeReader(reader1, writer1)
	buf4 := make([]byte, 40)
	for i := 0; i < 8; i++ {
		n, err = reader4.Read(buf4)
		executeIfNoErr(err, func() {
			fmt.Printf("Read(%d): %q\n", n, buf4[:n])
		})
	}
	fmt.Println()

	// Example 5.
	reader5a := strings.NewReader(
		"MultiReader returns a Reader that's the logical concatenation of " +
			"the provided input readers.")
	reader5b := strings.NewReader("They're read sequentially.")
	reader5c := strings.NewReader("Once all inputs have returned EOF, " +
		"Read will return EOF.")
	reader5d := strings.NewReader("If any of the readers return a non-nil, " +
		"non-EOF error, Read will return that error.")
	fmt.Println("New a multi-reader with 4 readers ...")
	reader5 := io.MultiReader(reader5a, reader5b, reader5c, reader5d)
	buf5 := make([]byte, 50)
	for i := 0; i < 8; i++ {
		n, err = reader5.Read(buf5)
		executeIfNoErr(err, func() {
			fmt.Printf("Read(%d): %q\n", n, buf5[:n])
		})
	}
	fmt.Println()

	// Example 6.
	fmt.Println("New a synchronous in-memory pipe ...")
	pReader, pWriter := io.Pipe()
	_ = interface{}(pReader).(io.ReadCloser)
	_ = interface{}(pWriter).(io.WriteCloser)

	comments := [][]byte{
		[]byte("Pipe creates a synchronous in-memory pipe."),
		[]byte("It can be used to connect code expecting an io.Reader "),
		[]byte("with code expecting an io.Writer."),
	}

	// This synchronization tool is added here to ensure that the print statements in the following example can be executed.
	// This is not necessary in practical use.
	var wg sync.WaitGroup
	wg.Add(2)

	go func() {
		defer wg.Done()
		for _, d := range comments {
			time.Sleep(time.Millisecond * 500)
			n, err := pWriter.Write(d)
			if err != nil {
				fmt.Printf("write error: %v\n", err)
				break
			}
			fmt.Printf("Written(%d): %q\n", n, d)
		}
		pWriter.Close()
	}()
	go func() {
		defer wg.Done()
		wBuf := make([]byte, 55)
		for {
			n, err := pReader.Read(wBuf)
			if err != nil {
				fmt.Printf("read error: %v\n", err)
				break
			}
			fmt.Printf("Read(%d): %q\n", n, wBuf[:n])
		}
	}()
	wg.Wait()
}

func executeIfNoErr(err error, f func()) {
	if err != nil {
		fmt.Printf("error: %v\n", err)
		return
	}
	f()
}

summary

We have been discussing and combing the program entities in the io code package today, especially those important interfaces and their implementation types.

The interface in io package plays an important role in the standard library of Go language and many third-party libraries. The core io.Reader interface and io.Writer interface are the extension objects or design sources of many interfaces. We will continue to explain the interface content in the IO package in the next section.

Note source code

https://github.com/MingsonZheng/go-core-demo