记一次重大的生产事故

高高兴兴地来上班，突闻任务大面积报错，经过一番调查，发现服务器上的某个用户不见了。

这是可坏的，某azkaban这个用户执行上面的所有任务，这个用户下面有很多crontab任务，全部都执行不了，包括给大boss发的邮件。

---

我的第一反应是快速添加用户，领导说等ldap自动同步，检查ldap服务，发现没问题，但连不上ldap服务器，网络丢包严重，等网络组处理。~
在此期间，由于网络问题，我使用其他用户执行脚本。

等啊等，等啊等…
运维哥等不及了，手动添加了这个用户，但脚本还是执行不了。报告如下错误:
Exception in thread “main” java.lang.RuntimeException: java.io.IOException: Permission denied
Caused by: java.io.IOException: Permission denied
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
查了下hive参数hive.exec.scratchdir设置目录:/tmp/hive
权限没有问题
然后查看hive日志/tmp/xx用户/hive.log，发现要创建本地目录

本地目录的用户和用户组都是数字，数字被删除uid和gid，虽然新用户的名称与被删除用户相同，但id不一样。
解决办法：
修改目录和文件的用户和组
当然，上述错误不一定是同样的原因，具体原因取决于日志。

---

至于用户为什么消失，运维给出的结论是网络异常，ldap服务器无法访问。我不太明白这一点。即使网络不通，用户也不会被删除，对吧？有可能被删除，无法同步。
Anyway，只要有人背锅，人为是否不那么重要。

PS:
我测试过，只要用户被删除，原本属于这个用户的文件就会变成id

被删前?? 被删后??

详细日志：
INFO [main] session.SessionState: Created HDFS directory: /tmp/hive/xx/28651db7-6f1b-4162-9ea4-cafe969c335b
INFO [main] session.SessionState: Created local directory: /hive/local/xx/28651db7-6f1b-4162-9ea4-cafe969c335b
INFO [main] session.SessionState: Created HDFS directory: /tmp/hive/xx/28651db7-6f1b-4162-9ea4-cafe969c335b/_tmp_space.db
INFO [main] conf.HiveConf: Using the default value passed in for log id: 28651db7-6f1b-4162-9ea4-cafe969c335b
INFO [main] session.SessionState: Updating thread name to 28651db7-6f1b-4162-9ea4-cafe969c335b main
INFO [28651db7-6f1b-4162-9ea4-cafe969c335b main] CliDriver: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

2022-04-20T21:26:21,302 INFO [21ec89d4-93b5-44e3-a8ba-0fc7ecfbbb46 main] CliDriver: Time taken: 4.602 seconds, Fetched: 1 row(s)
2022-04-20T21:26:21,302 INFO [21ec89d4-93b5-44e3-a8ba-0fc7ecfbbb46 main] conf.HiveConf: Using the default value passed in for log id: 21ec89d4-93b5-44e3-a8ba-0fc7ecfbbb46
2022-04-20T21:26:21,302 INFO [21ec89d4-93b5-44e3-a8ba-0fc7ecfbbb46 main] session.SessionState: Resetting thread name to main
2022-04-20T21:26:21,302 INFO [main] conf.HiveConf: Using the default value passed in for log id: 21ec89d4-93b5-44e3-a8ba-0fc7ecfbbb46
2022-04-20T21:26:21,306 INFO [main] session.SessionState: Deleted directory: /tmp/hive/xx/21ec89d4-93b5-44e3-a8ba-0fc7ecfbbb46 on fs with scheme hdfs
2022-04-20T21:26:21,311 INFO [main] session.SessionState: Deleted directory: /hive/local/xx/21ec89d4-93b5-44e3-a8ba-0fc7ecfbbb46 on fs with scheme file
2022-04-20T21:26:21,312 INFO [main] metastore.HiveMetaStore: 0: Cleaning up thread local RawStore…
2022-04-20T21:26:21,312 INFO [main] HiveMetaStore.audit: ugi=xx ip=unknown-ip-addr cmd=Cleaning up thread local RawStore…
2022-04-20T21:26:21,312 INFO [main] metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore
2022-04-20T21:26:21,312 INFO [main] HiveMetaStore.audit: ugi=xx ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
2022-04-21T00:00:14,722 INFO [main] metadata.Hive: Registering function row_seq com.bigdata.hive.udf.impl.HLSequenceGenerator
2022-04-21T00:00:14,723 WARN [main] metadata.Hive: Failed to register persistent function row_seq:com.bigdata.hive.udf.impl.HLSequenceGenerator. Ignore and continue.
2022-04-21T00:00:14,723 INFO [main] metadata.Hive: Registering function decrypt_hyd com.credithc.udf.DecryptHydDataFunction
2022-04-21T00:00:14,724 INFO [main] metadata.Hive: Registering function udfsharesc org.yjy.udfshare.udfsharesc
2022-04-21T00:00:14,725 INFO [main] metadata.Hive: Registering function fieldcrc32 com.cn.HiveUDF
2022-04-21T00:00:14,726 WARN [main] metadata.Hive: Failed to register persistent function fieldcrc32:com.cn.HiveUDF. Ignore and continue.
2022-04-21T00:00:14,726 INFO [main] metadata.Hive: Registering function decrypt_mobile com.credithc.udf.DecryptBdpDataFunction
2022-04-21T00:00:14,726 INFO [main] metadata.Hive: Registering function decrypt_mobile com.credithc.udf.DecryptBdpDataFunction
2022-04-21T00:00:14,727 INFO [main] metadata.Hive: Registering function parse_json com.hc.udf.JsonPar
2022-04-21T00:00:14,727 WARN [main] metadata.Hive: Failed to register persistent function parse_json:com.hc.udf.JsonPar. Ignore and continue.
202204-21T00:00:14,727 INFO [main] metadata.Hive: Registering function parse_json com.hc.udf.JsonPar
2022-04-21T00:00:14,728 WARN [main] metadata.Hive: Failed to register persistent function parse_json:com.hc.udf.JsonPar. Ignore and continue.
2022-04-21T00:00:14,728 INFO [main] metadata.Hive: Registering function getduration com.hc.HUGetDuration
2022-04-21T00:00:14,734 INFO [main] metadata.Hive: Registering function encrypt_bdp com.credithc.udf.EncryptBdpDataFunction
2022-04-21T00:00:14,734 INFO [main] metadata.Hive: Registering function addfive com.atguigu.myudf.MyUdf
2022-04-21T00:00:14,735 WARN [main] metadata.Hive: Failed to register persistent function addfive:com.atguigu.myudf.MyUdf. Ignore and continue.
2022-04-21T00:00:14,735 INFO [main] metadata.Hive: Registering function splitstr com.atguigu.myudf.MyUdf
2022-04-21T00:00:14,736 WARN [main] metadata.Hive: Failed to register persistent function splitstr:com.atguigu.myudf.MyUdf. Ignore and continue.
2022-04-21T00:00:14,736 INFO [main] metadata.Hive: Registering function splitstr2 com.atguigu.myudtf.MyUdtf
2022-04-21T00:00:14,736 WARN [main] metadata.Hive: Failed to register persistent function splitstr2:com.atguigu.myudtf.MyUdtf. Ignore and continue.