大数据培训之Fsimage和Edits解析

  1. 概念

大数据培训

  1. oiv查看Fsimage文件

(1)查看oiv和oev命令

[atguigu@hadoop102 current]$ hdfs

oiv            apply the offline fsimage viewer to an fsimage

oev            apply the offline edits viewer to an edits file

(2)基本语法

hdfs oiv -p 文件类型 -i镜像文件 -o 转换后文件输出路径

(3)案例实操

[atguigu@hadoop102 current]$ pwd

/opt/module/hadoop-2.7.2/data/tmp/dfs/name/current

 

[atguigu@hadoop102 current]$ hdfs oiv -p XML -i fsimage_0000000000000000025 -o /opt/module/hadoop-2.7.2/fsimage.xml

 

[atguigu@hadoop102 current]$ cat /opt/module/hadoop-2.7.2/fsimage.xml

将显示的xml文件内容拷贝到Eclipse中创建的xml文件中,并格式化。部分显示结果如下。

<inode>

  <id>16386</id>

  <type>DIRECTORY</type>

  <name>user</name>

  <mtime>1512722284477</mtime>

  <permission>atguigu:supergroup:rwxr-xr-x</permission>

  <nsquota>-1</nsquota>

  <dsquota>-1</dsquota>

</inode>

<inode>

  <id>16387</id>

  <type>DIRECTORY</type>

  <name>atguigu</name>

  <mtime>1512790549080</mtime>

  <permission>atguigu:supergroup:rwxr-xr-x</permission>

  <nsquota>-1</nsquota>

  <dsquota>-1</dsquota>

</inode>

<inode>

  <id>16389</id>

  <type>FILE</type>

  <name>wc.input</name>

  <replication>3</replication>

  <mtime>1512722322219</mtime>

  <atime>1512722321610</atime>

  <perferredBlockSize>134217728</perferredBlockSize>

  <permission>atguigu:supergroup:rw-r–r–</permission>

  <blocks>

     <block>

        <id>1073741825</id>

        <genstamp>1001</genstamp>

        <numBytes>59</numBytes>

     </block>

  </blocks>

</inode >

思考:可以看出,Fsimage中没有记录块所对应DataNode,为什么?

在集群启动后,要求DataNode上报数据块信息,并间隔一段时间后再次上报。

  1. oev查看Edits文件

(1)基本语法

hdfs oev -p 文件类型 -i编辑日志 -o 转换后文件输出路径

(2)案例实操

[atguigu@hadoop102 current]$ hdfs oev -p XML -i edits_0000000000000000012-0000000000000000013 -o /opt/module/hadoop-2.7.2/edits.xml

 

[atguigu@hadoop102 current]$ cat /opt/module/hadoop-2.7.2/edits.xml

将显示的xml文件内容拷贝到Eclipse中创建的xml文件中,并格式化。显示结果如下。

<?xml version=”1.0″ encoding=”UTF-8″?>

<EDITS>

  <EDITS_VERSION>-63</EDITS_VERSION>

  <RECORD>

     <OPCODE>OP_START_LOG_SEGMENT</OPCODE>

     <DATA>

        <TXID>129</TXID>

     </DATA>

  </RECORD>

  <RECORD>

     <OPCODE>OP_ADD</OPCODE>

     <DATA>

        <TXID>130</TXID>

        <LENGTH>0</LENGTH>

        <INODEID>16407</INODEID>

        <PATH>/hello7.txt</PATH>

        <REPLICATION>2</REPLICATION>

        <MTIME>1512943607866</MTIME>

        <ATIME>1512943607866</ATIME>

        <BLOCKSIZE>134217728</BLOCKSIZE>

        <CLIENT_NAME>DFSClient_NONMAPREDUCE_-1544295051_1</CLIENT_NAME>

        <CLIENT_MACHINE>192.168.1.5</CLIENT_MACHINE>

        <OVERWRITE>true</OVERWRITE>

        <PERMISSION_STATUS>

           <USERNAME>atguigu</USERNAME>

           <GROUPNAME>supergroup</GROUPNAME>

           <MODE>420</MODE>

        </PERMISSION_STATUS>

        <RPC_CLIENTID>908eafd4-9aec-4288-96f1-e8011d181561</RPC_CLIENTID>

        <RPC_CALLID>0</RPC_CALLID>

     </DATA>

  </RECORD>

  <RECORD>

     <OPCODE>OP_ALLOCATE_BLOCK_ID</OPCODE>

     <DATA>

        <TXID>131</TXID>

        <BLOCK_ID>1073741839</BLOCK_ID>

     </DATA>

  </RECORD>

  <RECORD>

     <OPCODE>OP_SET_GENSTAMP_V2</OPCODE>

     <DATA>

        <TXID>132</TXID>

        <GENSTAMPV2>1016</GENSTAMPV2>

     </DATA>

  </RECORD>

  <RECORD>

     <OPCODE>OP_ADD_BLOCK</OPCODE>

     <DATA>

        <TXID>133</TXID>

        <PATH>/hello7.txt</PATH>

        <BLOCK>

           <BLOCK_ID>1073741839</BLOCK_ID>

           <NUM_BYTES>0</NUM_BYTES>

           <GENSTAMP>1016</GENSTAMP>

        </BLOCK>

        <RPC_CLIENTID></RPC_CLIENTID>

        <RPC_CALLID>-2</RPC_CALLID>

     </DATA>

  </RECORD>

  <RECORD>

     <OPCODE>OP_CLOSE</OPCODE>

     <DATA>

        <TXID>134</TXID>

        <LENGTH>0</LENGTH>

        <INODEID>0</INODEID>

        <PATH>/hello7.txt</PATH>

        <REPLICATION>2</REPLICATION>

        <MTIME>1512943608761</MTIME>

        <ATIME>1512943607866</ATIME>

        <BLOCKSIZE>134217728</BLOCKSIZE>

        <CLIENT_NAME></CLIENT_NAME>

        <CLIENT_MACHINE></CLIENT_MACHINE>

        <OVERWRITE>false</OVERWRITE>

        <BLOCK>

           <BLOCK_ID>1073741839</BLOCK_ID>

           <NUM_BYTES>25</NUM_BYTES>

           <GENSTAMP>1016</GENSTAMP>

        </BLOCK>

        <PERMISSION_STATUS>

           <USERNAME>atguigu</USERNAME>

           <GROUPNAME>supergroup</GROUPNAME>

           <MODE>420</MODE>

        </PERMISSION_STATUS>

     </DATA>

  </RECORD>

</EDITS >

思考:NameNode如何确定下次开机启动的时候合并哪些Edits?

想要了解跟多关于大数据培训课程内容欢迎关注尚硅谷大数据培训,尚硅谷除了这些技术文章外还有免费的高质量大数据培训课程视频供广大学员下载学习


上一篇:
下一篇: