注册 登录  
 加关注
   显示下一条  |  关闭
温馨提示!由于新浪微博认证机制调整,您的新浪微博帐号绑定已过期,请重新绑定!立即重新绑定新浪微博》  |  关闭

和申的个人主页

专注于java开发,1985wanggang

 
 
 

日志

 
 

Using HDFS Programmatically  

2010-10-24 21:01:28|  分类: java分布 |  标签: |举报 |字号 订阅

  下载LOFTER 我的照片书  |
Using HDFS Programmatically - 和申 - 和申的个人主页

While HDFS can be manipulated explicitly through user commands, or implicitly as the input to or output from a Hadoop MapReduce job, you can also work with HDFS inside your own Java applications. (A JNI-based wrapper, libhdfs also provides this functionality in C/C++ programs.)

This section provides a short tutorial on using the Java-based HDFS API. It will be based on the following code listing:

1:  import java.io.File;  2:  import java.io.IOException;  3:  4:  import org.apache.hadoop.conf.Configuration;  5:  import org.apache.hadoop.fs.FileSystem;  6:  import org.apache.hadoop.fs.FSDataInputStream;  7:  import org.apache.hadoop.fs.FSDataOutputStream;  8:  import org.apache.hadoop.fs.Path;  9:  10: public class HDFSHelloWorld {  11:  12:   public static final String theFilename = "hello.txt";  13:   public static final String message = "Hello, world!\n";  14:  15:   public static void main (String [] args) throws IOException {  16:  17:     Configuration conf = new Configuration();  18:     FileSystem fs = FileSystem.get(conf);  19:  20:     Path filenamePath = new Path(theFilename);  21:  22:     try {  23:       if (fs.exists(filenamePath)) {  24:         // remove the file first  25:         fs.delete(filenamePath);  26:       }  27:  28:       FSDataOutputStream out = fs.create(filenamePath);  29:       out.writeUTF(message;  30:       out.close();  31:  32:       FSDataInputStream in = fs.open(filenamePath);  33:       String messageIn = in.readUTF();  34:       System.out.print(messageIn);  35:       in.close();  46:     } catch (IOException ioe) {  47:       System.err.println("IOException during operation: " + ioe.toString());  48:       System.exit(1);  49:     }  40:   }  41: }

This program creates a file named hello.txt, writes a short message into it, then reads it back and prints it to the screen. If the file already existed, it is deleted first.

First we get a handle to an abstract FileSystem object, as specified by the application configuration. The Configuration object created uses the default parameters.

17:     Configuration conf = new Configuration();  18:     FileSystem fs = FileSystem.get(conf);

The FileSystem interface actually provides a generic abstraction suitable for use in several file systems. Depending on the Hadoop configuration, this may use HDFS or the local file system or a different one altogether. If this test program is launched via the ordinary ‘java classname‘ command line, it may not find conf/hadoop-site.xml and will use the local file system. To ensure that it uses the proper Hadoop configuration, launch this program through Hadoop by putting it in a jar and running:

$HADOOP_HOME/bin/hadoop jar yourjar HDFSHelloWorld

Regardless of how you launch the program and which file system it connects to, writing to a file is done in the same way:

28:       FSDataOutputStream out = fs.create(filenamePath);  29:       out.writeUTF(message);  30:       out.close();

First we create the file with the fs.create() call, which returns an FSDataOutputStream used to write data into the file. We then write the information using ordinary stream writing functions; FSDataOutputStream extends the java.io.DataOutputStream class. When we are done with the file, we close the stream with out.close().

This call to fs.create() will overwrite the file if it already exists, but for sake of example, this program explicitly removes the file first anyway (note that depending on this explicit prior removal is technically a race condition). Testing for whether a file exists and removing an existing file are performed by lines 23-26:

23:       if (fs.exists(filenamePath)) {  24:         // remove the file first  25:         fs.delete(filenamePath);  26:       }

Other operations such as copying, moving, and renaming are equally straightforward operations on Path objects performed by the FileSystem.

Finally, we re-open the file for read, and pull the bytes from the file, converting them to a UTF-8 encoded string in the process, and print to the screen:

32:       FSDataInputStream in = fs.open(filenamePath);  33:       String messageIn = in.readUTF();  34:       System.out.print(messageIn);  35:       in.close();

The fs.open() method returns an FSDataInputStream, which subclasses java.io.DataInputStream. Data can be read from the stream using the readUTF() operation, as on line 33. When we are done with the stream, we call close() to free the handle associated with the file.

  评论这张
 
阅读(651)| 评论(0)
推荐 转载

历史上的今天

在LOFTER的更多文章

评论

<#--最新日志,群博日志--> <#--推荐日志--> <#--引用记录--> <#--博主推荐--> <#--随机阅读--> <#--首页推荐--> <#--历史上的今天--> <#--被推荐日志--> <#--上一篇,下一篇--> <#-- 热度 --> <#-- 网易新闻广告 --> <#--右边模块结构--> <#--评论模块结构--> <#--引用模块结构--> <#--博主发起的投票-->
 
 
 
 
 
 
 
 
 
 
 
 
 
 

页脚

网易公司版权所有 ©1997-2016