Introduction

In this tutorial, we’ll build a powerful downloader that allows downloading files from Google Drive and other cloud providers. With Golang’s efficient concurrency patterns, you'll be able to manage multiple downloads concurrently, stream large files, and track progress in real-time. Whether you’re downloading a few small files or handling large data sets, this project will showcase how to build a scalable and robust downloader that can easily be extended to support multiple cloud platforms.

If you're looking for a way to simplify and automate downloading large files, this tutorial is perfect for you. By the end, you’ll have a flexible and customizable Go-based downloader to suit your needs.

In a hurry?

If you're just looking to use this downloader with a UI, visit Go Downloader's Github. You'll find the docs to get it running fast.

What You’ll Learn

• Go Concurrency Patterns:
  Learn how to use Goroutines, channels and mutexes to handle multiple concurrent file downloads efficiently.

• Streaming Large Downloads:
  Explore how to stream large files while managing memory and system resources effectively.

• Concurrent File Downloads:
  Understand how to download files concurrently, speeding up the process and improving performance.

• Real-Time Progress Updates:
  Implement progress tracking to provide real-time feedback on download status.

• Handling Interruptions and Cancellations:
  Learn how to gracefully cancel one or all ongoing downloads.

Note: This tutorial will only focus on the core downloading logic.

Environment Setup

First before doing anything make sure to properly setup your environment to avoid potential bugs in future.

Prerequisites

• Install Go
• AIR for auto reloading
• Makefile to run complex commands
• Goose for PostgreSQL migrations

Configuring Makefile

Create a makefile at the root of the project with the following.

# Load environment variables from .env file
include ./.env
 
# To run the application
run: build
	@./bin/go-downloader
 
# Build the application
build:
	@go build -tags '!dev' -o bin/go-downloader
 
# Database migration status
db-status:
	@GOOSE_DRIVER=postgres GOOSE_DBSTRING=$(DB_URL) goose -dir=$(migrationPath) status
 
# Run database migrations
up:
	@GOOSE_DRIVER=postgres GOOSE_DBSTRING=$(DB_URL) goose -dir=$(migrationPath) up
 
# Roll back the last database migration
down:
	@GOOSE_DRIVER=postgres GOOSE_DBSTRING=$(DB_URL) goose -dir=$(migrationPath) down
 
# Reset database migrations
reset:
	@GOOSE_DRIVER=postgres GOOSE_DBSTRING=$(DB_URL) goose -dir=$(migrationPath) reset

High-Level Folder Structure Overview

go-downloader/
├── api
├── config
├── migrations
├── service
├── setting
├── store
├── types
├── util
├── .env
├── .air.toml
├── Makefile
├── go.mod
├── go.sum
└── main.go

Setting environment variables

Create a .env file in root or handle environment variables however you like, we'll use joho/godotenv package.

GOOGLE_CLIENT_ID
GOOGLE_CLIENT_SECRET
SESSION_SECRET=something-super-secret
APP_URL=http://localhost:3000
POSTGRES_USER
POSTGRES_PASSWORD
POSTGRES_DB

Creating a Web Server

We'll now start creating the web server that'll handle all the incoming requests. If you're not familiar with an API Server in go we recommend checking this guide for better clarity.

Heads Up! The main part of this guide begins here. Get ready to dive in!

API Layer

To start, create the following files inside api folder api.go and route.go

route.go file setup

All API routes will be defined in this here. We create a NewRouter struct that takes the env configuration, allowing all routes and handlers to access environment variables.

package api
 
import (
	"github.com/gofiber/fiber/v2"
	"github.com/nilotpaul/go-downloader/config"
)
 
type Router struct {
	env       config.EnvConfig
}
 
func NewRouter(env config.EnvConfig) *Router {
	return &Router{
		env:      env,
	}
}
 
func (h *Router) RegisterRoutes(r fiber.Router) {
	r.Get("/healthcheck", func(c *fiber.Ctx) error {
		return c.JSON("OK")
	})
}

api.go file setup

Here, we’ll add all the necessary middlewares, such as CORS and logging, before starting the server.

type APIServer struct {
	listenAddr string
	env        config.EnvConfig
}
 
func NewAPIServer(listenAddr string, env config.EnvConfig) *APIServer {
	return &APIServer{
		listenAddr: listenAddr,
		env:        env,
	}
}
 
func (s *APIServer) Start() error {
	app := fiber.New(fiber.Config{
		AppName:      "Go Downloader",
	})
 
	handler := NewRouter()
	handler.RegisterRoutes(app)
 
	log.Printf("Server started on http://localhost:%s", s.listenAddr)
 
	return app.Listen(":" + s.listenAddr)
}

Main Entrypoint

This is the main package in main.go file which will act as a entrypoint to the whole.

func main() {
	// Loads all Env vars from .env file.
	env := config.MustLoadEnv()
 
	log.Fatal(s.Start())
}

This is enough to start up the server and test it.

Start the server

air

that's it.😊

Testing

curl http://localhost:3000/healthcheck

The response should be OK with status 200

Creating a Provider Store

We need to implement a scalable solution for adding support for multiple cloud providers if necessary.

Working on provider registry

// Better to keep it in a seperate folder.
// Specific only to OAuth Providers.
type OAuthProvider interface {
	Authenticate(string) error
	GetAccessToken() string
	GetRefreshToken() string
	RefreshToken(*fiber.Ctx, string, bool) (*oauth2.Token, error)
	IsTokenValid() bool
	GetAuthURL(state string) string
	CreateOrUpdateAccount() (string, error)
	CreateSession(c *fiber.Ctx, userID string) error
	UpdateTokens(*GoogleAccount) error
}
 
type ProviderRegistry struct {
	Providers map[string]OAuthProvider
}
 
func NewProviderRegistry() *ProviderRegistry {
	return &ProviderRegistry{
		Providers: make(map[string]OAuthProvider),
	}
}
 
func (r *ProviderRegistry) Register(providerName string, p OAuthProvider) {
	r.Providers[providerName] = p
}
 
func (r *ProviderRegistry) GetProvider(providerName string) (OAuthProvider, error) {
	p, exists := r.Providers[providerName]
	if !exists {
		return nil, fmt.Errorf("Provider not found")
	}
 
	return p, nil
}

The ProviderRegistry serves as a central map to hold all our OAuth providers. When we initialize our providers, we'll register them in this map. This allows us to easily access any registered provider's functionalities throughout our service.

You'll see this action later.

Initializing the Provider Store

We'll register our providers based on the environment variables provided.

func InitStore(env config.EnvConfig) *ProviderRegistry {
	r := NewProviderRegistry()
 
	if len(env.GoogleClientSecret) != 0 || len(env.GoogleClientID) != 0 {
		googleProvider := NewGoogleProvider(googleProviderConfig{
			googleClientID:     env.GoogleClientID,
			googleClientSecret: env.GoogleClientSecret,
			googleRedirectURL:  env.AppURL + "/callback/google",
		}, env)
 
		r.Register("google", googleProvider)
	}
 
	return r
}

Using it in main.go

This will initialize all the cloud providers at startup using a singleton pattern, ensuring they are globally accessible throughout the application.

Here's a diagram explaining this concept:

r := store.InitStore(*env)

Database Setup

Create db.go in config

package config
 
import (
	"database/sql"
	"fmt"
	"log"
 
	_ "github.com/lib/pq"
)
 
func initDB(DBURL string) (*sql.DB, error) {
	db, err := sql.Open("postgres", DBURL)
	if err != nil {
		return nil, err
	}
 
	if err = db.Ping(); err != nil {
		return nil, err
	}
	log.Println("DB Connected")
    
	return db, nil
}
 
func MustInitDB(DBURL string) *sql.DB {
	db, err := initDB(DBURL)
	if err != nil {
		panic(fmt.Sprintf("failed to connect to DB: %s", err))
	}
 
	return db
}

Creating migrations

Check Goose Docs to see how to create database migrations.

CREATE TABLE IF NOT EXISTS "users" (
    id UUID PRIMARY KEY NOT NULL DEFAULT gen_random_uuid(),
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
 
CREATE TABLE IF NOT EXISTS "google_accounts" (
    id VARCHAR(255) PRIMARY KEY NOT NULL DEFAULT gen_random_uuid(),
    user_id UUID REFERENCES users(id) ON DELETE CASCADE,
    access_token VARCHAR(255) NOT NULL,
    refresh_token VARCHAR(255) NOT NULL,
    expires_at TIMESTAMP NOT NULL,
    token_type TEXT NOT NULL,  -- This was added in the second migration
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL
);

You can check if it's working by running make db-status

Run the following command to apply the database migrations:

make up

Google OAuth Setup

We'll need to authorize our application to access Google Drive using OAuth 2.0. This will allow us to download both public and private files from Google Drive while maintaining privacy.

Note: As this is too big to cover here, only essentials will be shown.

Installing OAuth2 Library

go get golang.org/x/oauth2

Working on Google Provider

type GoogleProvider struct {
	Config     *oauth2.Config
	db         *sql.DB
	env        config.EnvConfig
	Token      *oauth2.Token
	HttpClient *http.Client
}
 
type googleProviderConfig struct {
	googleClientID     string
	googleClientSecret string
	googleRedirectURL  string
}
 
var scopes = []string{
	"https://www.googleapis.com/auth/drive.readonly",
	"https://www.googleapis.com/auth/userinfo.email",
}
 
func NewGoogleProvider(cfg googleProviderConfig, db *sql.DB, env config.EnvConfig) *GoogleProvider {
	config := &oauth2.Config{
		ClientID:     cfg.googleClientID,
		ClientSecret: cfg.googleClientSecret,
		RedirectURL:  cfg.googleRedirectURL,
		Scopes:       scopes,
		Endpoint:     google.Endpoint,
	}
 
	return &GoogleProvider{
		Config: config,
		db:     db,
		env:    env,
	}
}

In the example above, we've set up a GoogleProvider struct, which holds the configuration and token information required for OAuth interactions with Google. This setup includes the OAuth2 configuration oauth2.Config, the database sql.DB, and environment variables config.EnvConfig.

While the core structure is in place, the next steps involve implementing key functions for:

• Generating the authorization URL to direct users for Google OAuth login page.
• Handling the callback after authorization to exchange the authorization code for an access token.
• Refreshing tokens to keep access persistent without requiring users to log in again frequently.

For this step refer to https://pkg.go.dev/golang.org/x/oauth2/google.

Wrapping Up

We’ve laid the groundwork for the Google Drive Downloader in Go, covering key components such as setting up the project structure, handling Google OAuth, and laying the foundation for future expansion. Along the way, we touched on some important topics:

• Database migrations with Goose to ensure smooth and scalable data management.
• Building a provider registry to easily extend support for additional cloud storage providers in the future.
• Designing a flexible architecture that makes it easy to handle more complex logic down the road.

That’s more than enough for one post, as things were getting pretty long! We’ll come back in Part 2 to finish our work, where we’ll work the main downloading functionality.

Until then, feel free to explore the current implementation in my GitHub and stay tuned for the next steps. Happy downloading!