Properties

Last built 1 year ago using go version go1.5.1 linux/amd64 and 39 seconds of time

Downloads
Filename
File Size
 
hdfs_master_darwin-386.zip
49KiB
hdfs_master_darwin-amd64.zip
49KiB
hdfs_master_linux-386.zip
49KiB
hdfs_master_linux-amd64.zip
49KiB
hdfs_master_linux-arm.zip
48KiB
hdfs_master_windows-386.zip
49KiB
hdfs_master_windows-amd64.zip
49KiB
Project-Readme for this version

HDFS for Go

GoDoc build

This is a native golang client for hdfs. It connects directly to the namenode using the protocol buffers API.

It tries to be idiomatic by aping the stdlib os package, where possible, and implements the interfaces from it, including os.FileInfo and os.PathError.

Here’s what it looks like in action:

client, _ := hdfs.New("namenode:8020")

file, _ := client.Open("/mobydick.txt")

buf := make([]byte, 59)
file.ReadAt(buf, 48847)

fmt.Println(string(buf))
// => Abominable are the tumblers into which he pours his poison.

For complete documentation, check out the Godoc.

The hdfs Binary

Along with the library, this repo contains a commandline client for HDFS. Like the library, its primary aim is to be idiomatic, by enabling your favorite unix verbs:

$ hdfs --help
Usage: ./hdfs COMMAND
The flags available are a subset of the POSIX ones, but should behave similarly.

Valid commands:
  ls [-lah] [FILE]...
  rm [-rf] FILE...
  mv [-fT] SOURCE... DEST
  mkdir [-p] FILE...
  touch [-amc] FILE...
  chmod [-R] OCTAL-MODE FILE...
  chown [-R] OWNER[:GROUP] FILE...
  cat SOURCE...
  head [-n LINES | -c BYTES] SOURCE...
  tail [-n LINES | -c BYTES] SOURCE...
  du [-sh] FILE...
  checksum FILE...
  get SOURCE [DEST]
  getmerge SOURCE DEST

Since it doesn’t have to wait for the JVM to start up, it’s also a lot faster hadoop -fs:

$ time hadoop fs -ls / > /dev/null

real  0m2.218s
user  0m2.500s
sys 0m0.376s

$ time hdfs ls / > /dev/null

real  0m0.015s
user  0m0.004s
sys 0m0.004s

Best of all, it comes with bash tab completion for paths!

Installing

To install the library, once you have Go all set up:

$ go get github.com/colinmarc/hdfs

Or, to install just the commandline client:

$ go get github.com/colinmarc/hdfs/cmd/hdfs

You’ll also want to add two lines to your .bashrc or .profile:

source $GOPATH/src/github.com/colinmarc/hdfs/cmd/hdfs/bash_completion
HADOOP_NAMENODE="namenode:8020"

Or, to install tab completion globally on linux:

ln -sT $GOPATH/src/github.com/colinmarc/hdfs/cmd/hdfs/bash_completion /etc/bash_completion.d/gohdfs

Acknowledgements

This library is heavily indebted to snakebite.