2016-09-09

Hbase修改表名

需要开启快照功能，在hbase-site.xml文件中添加如下配置项：

<property>
    <name>hbase.snapshot.enabled</name>
    <value>true</value>
</property>

进入hbase shell执行：

hbase shell> disable 'tableName'
hbase shell> snapshot 'tableName', 'tableSnapshot'
hbase shell> clone_snapshot 'tableSnapshot', 'newTableName'
hbase shell> delete_snapshot 'tableSnapshot'
hbase shell> drop 'tableName'

2016-09-01

elasticsearch2.3.1升级到2.3.5过程

由于在使用2.3.1版本elasticsearch过程中发现经常出现异常RemoteTransportException - AlreadyClosedException[this IndexReader is closed]，发现是在2.3.0中引入的一个bug，在2.3.3中进行了修复，所以对其进行了升级。在小版本之间升级可以进行滚动升级(Rolling upgrades)，但大版本之间就只能集群重启升级(Full cluster restart upgrade)。

2016-08-24

自动登录到linux机器

使用sshpass实现的一个自动登录脚本。

2016-08-24

一个有趣的canvas

发现一个有趣的canvas，可以改改给女朋友一个惊喜。

2016-08-23

如何在pg_dump时传入密码

在做postgres的备份时，需要定时执行pg_dump命令，这就需要在人不介入的情况传入密码，有两种方式可以实现。
一是使用环境变量PGPASSWORD，二是使用.pgpass文件。

2016-08-22

hive join遇到的问题及解决方法

这几天遇到一个问题，hive在执行join操作时报错，return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask，又没有特别有用的信息，经过一番分析和搜索找到解决方法。在这里记录一下。

2016-08-21

elasticsearch搜索示例

elasticsearch restful api和java api搜索示例。

2016-08-17

Memory Reserved on YARN

在yarn的resource manager页面中有一项是Memory Reserved，解释如下:

An implementation detail of this change that prevents applications from starving under this new flexibility is the notion of reserved containers. Imagine two jobs are running that each have enough tasks to saturate more than the entire cluster. One job wants each of its mappers to get 1GB, and another job wants its mappers to get 2GB. Suppose the first job starts and fills up the entire cluster. Whenever one of its task finishes, it will leave open a 1GB slot. Even though the second job deserves the space, a naive policy will give it to the first one because it’s the only job with tasks that fit. This could cause the second job to be starved indefinitely. To prevent this unfortunate situation, when space on a node is offered to an application, if the application cannot immediately use it, it reserves it, and no other application can be allocated a container on that node until the reservation is fulfilled. Each node may have only one reserved container. The total reserved memory amount is reported in the ResourceManager UI. A high number means that it may take longer for new jobs to get space.

参考http://stackoverflow.com/a/28567229

2016-08-15