hive on spark (Failed to create spark client)

hive on spark (Failed to create spark client)

janice No Comments
  未分类

执行hive的时候,发现Failed to create spark client.

没去检查日志,想了想,可能是hdfs的history文件有问题,上去hdfs创建相应的/user目录.问题解决.

后面检查了日志,果然发现了是这个目录权限问题..

hue 3.9.0提交coordinator bug

janice No Comments
  未分类

errorLog:

response = wrapped_callback(request, *callback_args, **callback_kwargs)
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/transaction.py”, line 371, in inner
return func(*args, **kwargs)
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/apps/oozie/src/oozie/decorators.py”, line 113, in decorate
return view_func(request, *args, **kwargs)
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/apps/oozie/src/oozie/decorators.py”, line 75, in decorate
return view_func(request, *args, **kwargs)
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/apps/oozie/src/oozie/views/editor2.py”, line 674, in submit_coordinator
job_id = _submit_coordinator(request, coordinator, mapping)
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/apps/oozie/src/oozie/views/editor2.py”, line 700, in _submit_coordinator
wf_dir = Submission(request.user, wf, request.fs, request.jt, mapping, local_tz=coordinator.data[‘properties’][‘timezone’]).deploy()
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/desktop/libs/liboozie/src/liboozie/submission2.py”, line 298, in deploy
oozie_xml = self.job.to_xml(self.properties)
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/apps/oozie/src/oozie/models2.py”, line 468, in to_xml
‘workflow_mapping’: workflow_mapping
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/desktop/core/src/desktop/lib/django_mako.py”, line 114, in render_to_string_normal
result = template.render(**data_dict)
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/build/env/lib/python2.7/site-packages/Mako-0.8.1-py2.7.egg/mako/template.py”, line 443, in render
return runtime._render(self, self.callable_, args, data)
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/build/env/lib/python2.7/site-packages/Mako-0.8.1-py2.7.egg/mako/runtime.py”, line 786, in _render
**_kwargs_for_callable(callable_, data))
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/build/env/lib/python2.7/site-packages/Mako-0.8.1-py2.7.egg/mako/runtime.py”, line 818, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/build/env/lib/python2.7/site-packages/Mako-0.8.1-py2.7.egg/mako/runtime.py”, line 844, in _exec_template
callable_(context, *args, **kwargs)
File “/tmp/tmpo3mDmS/oozie/editor2/gen/workflow.xml.mako.py”, line 110, in render_body
__M_writer( node.to_xml(mapping, node_mapping, workflow_mapping) )
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/apps/oozie/src/oozie/models2.py”, line 860, in to_xml
notebook = Notebook(document=Document2.objects.get_by_uuid(user=self.user, uuid=self.data[‘properties’][‘uuid’]))
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/desktop/core/src/desktop/models.py”, line 975, in get_by_uuid
latest_doc.can_read_or_exception(user)
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/desktop/core/src/desktop/models.py”, line 1305, in can_read_or_exception
if self.can_read(user):
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/desktop/core/src/desktop/models.py”, line 1301, in can_read
has_read_permissions = perm.user_has_access(user) if perm else False
File “/opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hue/desktop/core/src/desktop/models.py”, line 1540, in user_has_access
return self.groups.filter(id__in=user.groups.all()).exists() or user in self.users.all()
AttributeError: ‘NoneType’ object has no attribute ‘groups’


lib/hue/apps/oozie/src/oozie/models2.py

第3404  以及 3409行,创建Workflow的时候没有传入user.导致验证失败.

修改源码

wf = Workflow(data=wf_doc.data, user=self.document.owner)


在添加sqoop任务的时候,单引号会出问题.用双引号.

bash分割数组的坑

janice No Comments
  未分类

TEST_DATA=”1/2/3″

OLD_IFS=$IFS

$IFS=”/”

LISTS=`ls /`

for i in $LISTS:

这时候其实已经按照”/”对LISTS进行了分割.会出错,IFS是一个全局的变量,所以要考虑一些函数的影响.慎用..

linux rename

janice No Comments
  未分类

Linux下批量修改文件名(rename)

原文地址: http://blog.csdn.net/sea_shore/article/details/6102437

1.rename命令批量修改文件名, 其实linux下可以使用别的办法来批量修改文件名, 不过rename实在太方便了

比如把所有的表为cdb1_* 修改为cdb_*的
在本目录下
只需要# rename ‘cdb1’ ‘cdb’ *

以前都是写个for循环来做…想想多傻啊, 呵呵

rename还有更多的功能, 建议man rename下

From:http://www.hao32.com/unix-linux/42.html

2.批量更改文件名 rename

通过 man rename 命令可以得知,rename 命令其实是一个 Perl 脚本命令,
它专用于批量地给多文件重命名(rename multiple files)。

命令格式:
rename [ -v ] [ -n ] [ -f ] perlexpr [ files ]
perlexpr 是一种 Perl 脚本格式的正则表达式。
参数:
-v, –verbose
Verbose: print names of files successfully renamed.
详细模式:打印成功更改的文件名列表
-n, –no-act
No Action: show what files would have been renamed.
测试模式:并不真正的执行命令,而只是显示哪些文件名应该怎么进行
更改,用于测试模式。
-f, –force
Force: overwrite existing files.
强制模式:在更改文件名,如果更改后的文件已经存在时覆盖已经存在
的文件。

rename 的典型应用:

0. 批量更改文件扩展名 
$ ls
1.txt 2.txt 3.txt 4.txt
$ rename ‘s//.txt//.ext/’ *
$ ls
1.ext 2.ext 3.ext 4.ext
1. 批量删除文件扩展名 
$ ls
1.txt 2.txt 3.txt 4.txt
$ rename ‘s//.txt//’ *
$ ls
1 2 3 4
2. 批量添加文件扩展名 
$ ls
1 2 3 4
$ rename ‘s/$//.txt/’ *
$ ls
1.txt 2.txt 3.txt 4.txt
3. 按自己的方式批量重命名文件 
$ ls
1.ext 2.ext 3.ext 4.ext
$ rename ‘s/(/d)/第$1 章/’ *
$ ls
第 1 章.ext 第 2 章.ext 第 3 章.ext 第 4 章.ext

3.

方法1:把文件名拆分处理,可以任意修改文件名
find -name ‘*.log’ -printf %f//n|awk -F’.’ ‘{print $1}’|xargs -i{} mv {}.log xiyun_{}.log
方法2:利用rename
一般的linux下的rename命令比较简单
rename ‘test’ ‘xiyun’ ‘*.log’
把文件名中的某部分字符串替换掉
ubuntu下的rename命令支持正则表达式,因此功能更强大。

方法3:直接利用find和xargs和mv,中间不用awk,这样只能添加后缀名,不能修改文件名。

From:http://blog.chinaunix.net/u/22583/showart_1964649.html


redhat 版本  rename .log .log22 *

 

 

flume相关

janice No Comments
  未分类

flume主要有source  channel  sink

source —>  channel  —–>sink

流程粗略如上图.当然,source可以流入多个channel,   sink也可以从多个channel里面拉取数据.

source到channel会有  interceptor,可以过滤事件并添加一些数据进header.

之后会有一个selector.selector根据控制source具体要流到哪个channel.


1、selector

选择器可以工作在复制 多路复用(路由) 模式下
复制模式
属性说明:
selector.type replicating
selector.optional     多路复用(路由)模式
属性说明:
selector.type    multiplexing
selector.header    指定要监测的头的名称
selector.default    –
selector.mapping.*    –
栗子:
t1.sources = r1
t1.channels = c1 c2 c3 c4
t1.sources.r1.selector.type = multiplexing
t1.sources.r1.selector.header = country
t1.sources.r1.selector.mapping.CN = c1
t1.sources.r1.selector.mapping.US= c2 c3
t1.sources.r1.selector.default = c4

分类目录

关于我

 Janice 詹

QQ   455899417

janice2014@foxmail.com