Lesson 9/41 · Go Language Intro & Dev Environment Beginner m01-l09

Module 1 Project: A CLI Word Counter mini-wc

Module 1: Go Language Intro & Dev Environment | Lesson: m01-l09 | Level: Beginner After this lesson you will be able to: combine Go's basic syntax and standard library to build, on your own, a testable command-line tool that supports multiple flags.

Learning Objectives

Understand the requirements analysis of a command-line tool, and master the design and use of the flag package.
Skillfully use bufio.Scanner for streaming file reading and standard input processing.
Use map[string]int for word-frequency counting, and the sort package for sorted output.
Split core logic into testable functions and handle common I/O errors.

Core Concepts

2.1 Requirements Analysis and Tool Design

Before building a command-line interface (CLI) tool, it is crucial to nail down the requirements. Our tool mini-wc is modeled on the classic Unix wc utility, with these main features: 1. Count lines: count the newline characters in the input. 2. Count words: count whitespace-separated words. 3. Count bytes: count the total bytes of the input (note: len(s) returns the byte count, not the character count. If you need the Unicode character count, use utf8.RuneCountInString; but this lesson targets wc -c, so we count bytes). 4. Word frequency: count how many times each word appears, and support sorting to output the Top N.

Design principles: - Flexible input source: support reading a given file; if no file is given, read from standard input (stdin). - Configurable flags: control the output (lines, words, bytes, word frequency) through command-line flags. - Testable core logic: separate I/O from business logic; the core counting functions should accept an io.Reader interface for easy unit testing.

2.2 Core Data Structures and Algorithm

Word-frequency counting is the core business. In Go, we use a map[string]int to store words and their counts. - Initialization: when a Go map is accessed with a missing key, it returns the zero value (the zero value of int is 0), so in a counting scenario no explicit initialization is needed; just m[word]++. - Sorting: maps are unordered. To output the Top N word frequencies, convert the map to a slice and sort it with sort.Slice.

File reading: - For large files or standard input, bufio.Scanner is recommended. It provides streaming processing with constant memory usage, instead of loading everything at once like os.ReadFile. - By default, bufio.Scanner caps a single line at 64KB. On a line that is too long, it returns bufio.ErrTooLong. In this lesson's example, we use scanner.Buffer to enlarge the buffer and support larger input.

2.3 CLI Flag Design and Function Splitting

Use the flag package to parse command-line arguments. To keep main concise and improve testability, we split the logic into: 1. CountStats(r io.Reader) (Stats, error): reads the input and counts lines, words and bytes. 2. CountWords(r io.Reader) ([]WordFreq, error): counts word frequency and returns a sorted slice. 3. main(): parses flags, opens the file / handles stdin, calls the functions above, and formats the output.

This split lets CountStats and CountWords be unit-tested directly by passing in a strings.NewReader, without touching a real file.

Example Code

Example 1: A Basic Word Counter

This example shows how to read a file, count lines, words and bytes, and print simple statistics.

package main

import (
    "bufio"
    "fmt"
    "os"
    "strings"
)

// Stats holds basic statistics
type Stats struct {
    Lines   int
    Words   int
    Bytes   int
}

func main() {
    if len(os.Args) < 2 {
        fmt.Fprintln(os.Stderr, "Usage: mini-wc <filename>")
        os.Exit(1)
    }

    filename := os.Args[1]
    file, err := os.Open(filename)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error: %v\n", err)
        os.Exit(1)
    }
    defer file.Close()

    stats, err := CountStats(file)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Error: %v\n", err)
        os.Exit(1)
    }

    fmt.Printf("%d %d %d %s\n", stats.Lines, stats.Words, stats.Bytes, filename)
}

// CountStats counts lines, words and bytes
func CountStats(r io.Reader) (Stats, error) {
    var stats Stats
    scanner := bufio.NewScanner(r)
    // Set a larger buffer to support long lines
    buf := make([]byte, 0, 64*1024)
    scanner.Buffer(buf, 1024*1024)

    for scanner.Scan() {
        stats.Lines++
        line := scanner.Text()
        stats.Bytes += len(line) + 1 // +1 for the newline character
        // Split words with strings.Fields, which handles multiple spaces automatically
        words := strings.Fields(line)
        stats.Words += len(words)
    }

    if err := scanner.Err(); err != nil {
        return stats, err
    }

    return stats, nil
}

Expected output:

3 15 78 sample.txt

(Note: this assumes sample.txt contains "hello world\nfoo bar baz\nGo is great"; the exact numbers depend on the file contents.)

Example 2: An Enhanced Version with Multiple Flags and Word Frequency

This example shows the complete mini-wc implementation, supporting the -l, -w, -c and -top flags, and handling either a file or standard input.

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
    "sort"
    "strings"

    "flag"
)

// Stats holds basic statistics
type Stats struct {
    Lines   int
    Words   int
    Bytes   int
}

// WordFreq holds a word and its count
type WordFreq struct {
    Word  string
    Count int
}

func main() {
    var countLines, countWords, countBytes, showTop bool
    var topN int

    flag.BoolVar(&countLines, "l", false, "count lines")
    flag.BoolVar(&countWords, "w", false, "count words")
    flag.BoolVar(&countBytes, "c", false, "count bytes")
    flag.BoolVar(&showTop, "top", false, "show top N word frequencies")
    flag.IntVar(&topN, "n", 5, "number of top words to show")

    flag.Parse()

    // Default behavior: if no flags are given, show all statistics
    if !countLines && !countWords && !countBytes && !showTop {
        countLines = true
        countWords = true
        countBytes = true
        showTop = false
    }

    var reader io.Reader
    var filename string

    if flag.NArg() > 0 {
        filename = flag.Arg(0)
        file, err := os.Open(filename)
        if err != nil {
            fmt.Fprintf(os.Stderr, "Error: %v\n", err)
            os.Exit(1)
        }
        defer file.Close()
        reader = file
    } else {
        reader = os.Stdin
        // If reading from stdin and word frequency is requested, the input must be cached,
        // because a Scanner can only be consumed once. To keep the example simple,
        // we assume word frequency requires a file, or that the user pipes in the input.
        if showTop {
            fmt.Fprintln(os.Stderr, "Error: -top requires a file input for re-reading, or use a pipe with caching.")
            os.Exit(1)
        }
    }

    var stats Stats
    var wordFreqs []WordFreq
    var err error

    if countLines || countWords || countBytes {
        stats, err = CountStats(reader)
        if err != nil {
            fmt.Fprintf(os.Stderr, "Error: %v\n", err)
            os.Exit(1)
        }
    }

    if showTop {
        // Reopen the file for word-frequency counting, because reader may already be consumed
        if filename != "" {
            file, err := os.Open(filename)
            if err != nil {
                fmt.Fprintf(os.Stderr, "Error: %v\n", err)
                os.Exit(1)
            }
            defer file.Close()
            wordFreqs, err = CountWords(file)
            if err != nil {
                fmt.Fprintf(os.Stderr, "Error: %v\n", err)
                os.Exit(1)
            }
            // Keep only the top N
            if topN < len(wordFreqs) {
                wordFreqs = wordFreqs[:topN]
            }
        }
    }

    // Print the results
    if countLines {
        fmt.Printf("%d\n", stats.Lines)
    }
    if countWords {
        fmt.Printf("%d\n", stats.Words)
    }
    if countBytes {
        fmt.Printf("%d\n", stats.Bytes)
    }
    if showTop {
        for _, wf := range wordFreqs {
            fmt.Printf("%s: %d\n", wf.Word, wf.Count)
        }
    }
}

// CountStats counts lines, words and bytes
func CountStats(r io.Reader) (Stats, error) {
    var stats Stats
    scanner := bufio.NewScanner(r)
    buf := make([]byte, 0, 64*1024)
    scanner.Buffer(buf, 1024*1024)

    for scanner.Scan() {
        stats.Lines++
        line := scanner.Text()
        stats.Bytes += len(line) + 1 // +1 for the newline character
        words := strings.Fields(line)
        stats.Words += len(words)
    }

    if err := scanner.Err(); err != nil {
        return stats, err
    }

    return stats, nil
}

// CountWords counts word frequency and returns a sorted slice
func CountWords(r io.Reader) ([]WordFreq, error) {
    wordCount := make(map[string]int)
    scanner := bufio.NewScanner(r)
    buf := make([]byte, 0, 64*1024)
    scanner.Buffer(buf, 1024*1024)

    for scanner.Scan() {
        line := scanner.Text()
        words := strings.Fields(line)
        for _, word := range words {
            // Simple approach: lowercase to unify counting.
            // In a real project you might need more sophisticated cleaning.
            word = strings.ToLower(word)
            wordCount[word]++
        }
    }

    if err := scanner.Err(); err != nil {
        return nil, err
    }

    // Convert to a slice so it can be sorted
    pairs := make([]WordFreq, 0, len(wordCount))
    for word, count := range wordCount {
        pairs = append(pairs, WordFreq{Word: word, Count: count})
    }

    // Sort by frequency, descending
    sort.Slice(pairs, func(i, j int) bool {
        return pairs[i].Count > pairs[j].Count
    })

    return pairs, nil
}

Expected output:

# Assuming sample.txt contains "hello world\nfoo bar baz\nGo is great"
# Run: go run example2.go -l -w -c sample.txt
3
15
78

Common Mistakes & Pitfalls

Using strings.Split instead of strings.Fields
- Wrong code: strings.Split(line, " ")
- Why: Split keeps empty strings. If a line contains several consecutive spaces, Split produces empty strings "", leading to empty words in the frequency count.
- Correct approach: use strings.Fields(line), which automatically handles runs of whitespace and filters out empty strings.
Modifying a map's structure while iterating it
- Wrong assumption: thinking that deleting a key during for range over a map causes a panic.
- Facts: in Go, deleting a key during iteration is legal; entries not yet reached that are deleted are no longer produced. But whether a newly added entry is produced is unspecified (the spec allows either).
- Correct approach: if you need to delete, just delete(m, key); if you need to add, do it after the loop, or make sure you do not rely on newly added entries being produced.
Ignoring bufio.Scanner's ErrTooLong error
- Wrong code: calling scanner.Scan() without checking scanner.Err().
- Why: by default, Scanner caps a line at 64KB. On an overlong line, Scan() returns false and Err() returns bufio.ErrTooLong. If you do not check, the program silently fails and the statistics are wrong.
- Correct approach: always check scanner.Err() after the loop, and enlarge the buffer as needed.

Best Practices

Separate core logic from I/O
- Encapsulate the counting logic in a function that takes an io.Reader, rather than operating on *os.File directly. This improves reuse and makes unit testing trivial (use strings.NewReader).
Use the flag package for arguments
- Do not hand-parse os.Args. The flag package gives you type safety, automatic help text and default values; it is the standard way to build CLI tools.
Push error handling up
- Core functions (such as CountStats) should return error, and let main handle error printing and exiting uniformly. Avoid calling os.Exit inside core functions: it skips defer cleanup and makes testing harder.
Note the difference between bytes and characters
- len(s) returns the number of bytes. If you need the character count (Unicode code points), use utf8.RuneCountInString(s). In wc-like tools, -c usually means bytes and -m means characters; keep them distinct.

Exercises

Concept question: explain why, in CountWords, we use map[string]int instead of explicitly initializing and then assigning. What advantage does Go's map zero-value behavior offer here?
Coding exercise: write a function FilterWords(freqs []WordFreq, minCount int) []WordFreq that takes a frequency slice and a minimum count threshold, and returns the words whose count is at least the threshold.
Advanced/debugging exercise: the code below tries to delete words with count 1 while iterating the map, but has a logic flaw. Find the problem and fix it. go func RemoveSingleWords(m map[string]int) map[string]int { for word, count := range m { if count == 1 { delete(m, word) } } return m }

Reference Answers

Reference Answers (click to expand / collapse)

Question 1

Answer: when a Go map is accessed with a missing key, it returns the zero value of the value type (the zero value of int is 0), so in a counting scenario no explicit initialization is needed. Just m[word]++: if the key does not exist, it first returns 0 and then adds 1, giving 1. If you need to distinguish "absent" from "present with value 0", use the v, ok := m[k] idiom.

Question 2

package main

import (
    "fmt"
)

type WordFreq struct {
    Word  string
    Count int
}

func FilterWords(freqs []WordFreq, minCount int) []WordFreq {
    var filtered []WordFreq
    for _, wf := range freqs {
        if wf.Count >= minCount {
            filtered = append(filtered, wf)
        }
    }
    return filtered
}

func main() {
    freqs := []WordFreq{
        {Word: "hello", Count: 5},
        {Word: "world", Count: 2},
        {Word: "go", Count: 1},
    }
    filtered := FilterWords(freqs, 2)
    for _, wf := range filtered {
        fmt.Printf("%s: %d\n", wf.Word, wf.Count)
    }
}

Explanation: the function iterates over the input slice, selects the elements that meet the condition, and returns a new slice, leaving the original untouched.

Question 3

Problem analysis: the original code calls delete(m, word) while iterating the map with for range. Although this is legal in Go and does not panic, the semantics of deleting keys during iteration are: entries not yet reached that are deleted are no longer produced. If the map is large, deletion may affect the determinism of iteration (though it does not panic). More seriously, if you need to do further work based on the remaining elements after deletion, this in-place modification can lead to muddled logic. In addition, if you need to re-sort or process after deleting, you may need to collect the keys to delete first.

Suggested fix: although the original code behaves deterministically in a single goroutine (deleted keys are no longer produced), for clarity and to avoid the potential overhead (internal map restructuring), collect the keys to delete first and then delete them in one pass. Or, if your goal is simply to filter, create a new map.

func RemoveSingleWords(m map[string]int) map[string]int {
    // Build a new map to avoid modifying the original during iteration
    newM := make(map[string]int)
    for word, count := range m {
        if count > 1 {
            newM[word] = count
        }
    }
    return newM
}

Explanation: this approach filters by building a new map, keeping the semantics clear and avoiding the risk of mutating a map during iteration.

Lesson Summary

You have applied the flag package to a CLI tool, achieving flexible argument parsing.
You have used bufio.Scanner for efficient, low-memory streaming reads.
You understand the central role of map[string]int in word-frequency counting, and have used the sort package for sorting.
You have practiced the design pattern of separating core logic from I/O, improving testability.

Next lesson preview: Module 2 begins — composite data types and object-oriented design: pointers and memory, structs and methods, interfaces, and more.