Android下的配置管理之道之OpenGrok代码索引环境搭建[通俗易懂]

You need the following:
JDK 1.8 or higher  java 版本需要1.8或者更高的
OpenGrok '''binaries''' from https://github.com/OpenGrok/OpenGrok/releases (.tar.gz file with binaries, not the source code tarball !)下载的包要是二进制的包，不是源码包，源码包你还要编译的。
 
 
https://github.com/universal-ctags for analysis (avoid Exuberant ctags, they are not maintained anymore) 
需要 universal-ctags 命令的 (ctags貌似有2个流派吧. Exuberant Ctags  和 ctags-universal )
 
A servlet container like GlassFish or Tomcat 8.0 or later also running with Java at least 1.8
需要servlet容器，推荐，也是我们常用的是tomcat，目前可以使用tomcat8了。或者tomcat 使用docker来 docker pull tomcat:9.0
If history is needed, appropriate binaries (in some cases also cvs/svn repository) must be present on the system (e.g. Subversion or Mercurial or SCCS or ... )
如果浏览代码索引的时候，想看代码的 历史记录。例如代码是git管理的，你的环境也要安装git的。
 
2GB of memory for the indexing process (bigger deployments will need more)
内存比较消耗的，代码很多的情况内存要大。要设置-Xmx参数的.
a recent browser for clients - IE, Firefox, recent Chrome or Safari
Optional tuning (see https://github.com/oracle/opengrok/wiki/Tuning-for-large-code-bases)
一个客户端浏览器。IE浏览器，火狐浏览器，谷歌浏览器，Safari浏览器。这不是废话嘛.......

OpenGrok 是基于 JAVA 的, 因此我们首先需要 JDK 和 JRE 来支持其运行

新版本的opengrok需要jdk8了

Latest Java 1.8

#Ubuntu14.04 的源中已经提供了Tomcat 7 的包

sudo apt-get install tomcat7

# 如果是16.04 可以安装tomcat8了。

从https://github.com/OpenGrok/OpenGrok/releases 下载最新的安装包，注意下载的是二进制包
而不是源码包。解压放到固定位置上。
 
 
例如 安装到 /opengrok 目录下面. 类似于opengrok的home目录了.
 
首选新建几个目录
mkdir /opengrok/{src,data,dist,etc,log}  # The indexer will need to write to /opengrok/{data,log} and the web application will need to be able to read from /opengrok/{src,etc,data}
 
解压安装包
tar -C /opengrok/dist --strip-components=1 -xzf opengrok-X.Y.Z.tar.gz
 
配置logger文件
cp /opengrok/dist/doc/logging.properties /opengrok/etc
 
 
下载android代码
cd /opengrok/src
git clone https://github.com/githubtraining/hellogitworld.git
git clone https://github.com/OpenGrok/OpenGrok
 
 
安装indexer相关工具. 采用第二种方式比较好.
第1种方式,
  python3 -m pip install opengrok_tools.tar.gz
第2种方式,
  cd /opt/opengrok
  python3 -m venv opengrok-tools
  opengrok-tools/bin/python -m pip install opengrok_tools.tar.gz
 
 
部署 source.war 到 tomcat 下面
opengrok-deploy -c /opengrok/etc/configuration.xml \
    /opengrok/dist/lib/source.war /var/lib/tomcat8/webapps
例如:
opengrok-deploy  -c dat_root/ctags/etc/configuration.xml source.war tomcat/webapps/ctags.war 
 
创建索引
opengrok-indexer \
    -J=-Djava.util.logging.config.file=/opengrok/etc/logging.properties \
    -a /opengrok/dist/lib/opengrok.jar -- \
    -c /usr/local/bin/ctags \
    -s /opengrok/src -d /opengrok/data -H -P -S -G \
    -W /opengrok/etc/configuration.xml -U http://localhost:8080/source
 
or
java \
    -Djava.util.logging.config.file=/opengrok/etc/logging.properties \
    -jar /opengrok/dist/lib/opengrok.jar \
    -c /usr/local/bin/ctags \
    -s /opengrok/src -d /opengrok/data -H -P -S -G \
    -W /opengrok/etc/configuration.xml -U http://localhost:8080/source
 
例如:
opengrok-indexer \
    -J=-Djava.util.logging.config.file=opengrok/doc/logging.properties \
    -a opengrok/lib/opengrok.jar -- \
    -s src_root/ctags -d dat_root/ctags -H -P -S -G \
    -W dat_root/ctags/configuration.xml -U http://localhost:8080/ctags 
 
最后设置 定时 更新代码, 更新索引

# 部署的
cat bin/deploy.sh 
#!/bin/bash -x
export OPENGROK_HOME=/home/test/opengrok
export PATH=$PATH:$OPENGROK_HOME/opengrok-tools/bin
export WORK=/work/test
export SITE=${1:-"sm8250"}
 
opengrok-deploy -c $WORK/data_root/$SITE/configuration.xml $WORK/bin/source.war $WORK/web_root/${SITE}.war 
sleep 30
n=0
until [ $n -ge 5 ]
do
   curl -sSf http://www.opengrok.com/$SITE && break  # substitute your command here
   n=$[$n+1]
   sleep 15
done
 
 
 
# 新建索引
test@xref:/work/test$ cat bin/indexer.sh 
#!/bin/bash -x
export OPENGROK_HOME=/home/test/opengrok
export PATH=$PATH:$OPENGROK_HOME/opengrok-tools/bin
export WORK=/work/test
export SITE=${1:-"sm8250"}
opengrok-indexer \
  -J=-Xmx16g -J=-server -J=-Djava.util.logging.config.file=$OPENGROK_HOME/etc/logging.properties \
  -a $OPENGROK_HOME/dist/opengrok-1.3.11/lib/opengrok.jar -- \
  -s $WORK/src_root/${SITE} -d $WORK/data_root/${SITE} -H -P -S -G \
  -W $WORK/data_root/${SITE}/configuration.xml -U http://www.opengrok.com/${SITE}

先 虚拟机上安装系统， 按照上面的把 工具解压好


备份整个系统, 然后使用docker import 导入.
tar -cvpzf backup.tar.gz --exclude=/backup.tar.gz --exclude=/proc   --exclude=/tmp  --exclude=/mnt  --exclude=/dev  --exclude=/sys  --exclude=/run  --exclude=/media    --exclude=/var/log  --exclude=/var/cache/apt/archives   --exclude=/usr/src/linux-headers*   --exclude=/home/*/.gvfs     --exclude=/home/*/.cache    --exclude=/home/*/.local/share/Trash 


导入到docker

/usr/bin/git log --abbrev-commit --abbrev=8 --name-only --pretty=fuller --date=iso8601-strict -m -- /src_root/frameworks/base
 
这个命令 执行非常慢

docker run --restart always -d --network host -p 8080:8080 --name tomcat9 -e JAVA_OPTS='-Xmx64g' \
    -v /work:/work \
    -v /work/test/web_root:/usr/local/tomcat/webapps \
    -v /home/test:/home/test \
    -v /etc/timezone:/etc/timezone:ro \
    -v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime:ro \
    harbor.test.com/ubuntu/tomcat9-jdk11:v1
 
改为这样，可以80和8080 同时访问。--network host 和 -p 不能同时使用。
docker run --restart always -d -p 80:8080 -p 8080:8080 --name tomcat9 -e JAVA_OPTS='-Xmx128g' \
    -v /work:/work \
    -v /home:/home \
    -v /work/test/web_root:/usr/local/tomcat/webapps \
    -v /etc/timezone:/etc/timezone:ro \
    -v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime:ro \
    harbor.test.com/ubuntu/tomcat9-jdk11:v1

+ export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home//opengrok-1.5.12/tools/opengrok-tools/bin
+ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/test/opengrok-1.5.12/tools/opengrok-tools/bin
+ opengrok-indexer -J=-Xmx128g -J=-server -a /home/test/opengrok-1.5.12/lib/opengrok.jar -- -s /work/test/src_root/ -d /work/test/data_root/ --verbose --progress -H -P -S --depth 1024 -W /work/test/data_root//configuration.xml -U http://localhost:8080//
Jun 04, 2021 9:39:15 AM org.opengrok.indexer.index.Indexer parseOptions
INFO: Indexer options: [-s, /work/test/src_root/, -d, /work/test/data_root/, --verbose, --progress, -H, -P, -S, --depth, 1024, -W, /work/test/data_root//configuration.xml, -U, http://localhost:8080//]
Logging filehandler pattern: %h/java%u.log
Jun 04, 2021 9:39:16 AM org.opengrok.indexer.index.Indexer main
SEVERE: Couldn't notify the webapp on http://localhost:8080//.
javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized
    at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:936)
    at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:739)
    at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:623)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:621)
    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:404)
    at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:300)
    at org.opengrok.indexer.index.IndexerUtil.enableProjects(IndexerUtil.java:79)
    at org.opengrok.indexer.index.Indexer.main(Indexer.java:334)
 
Couldn't notify the webapp on http://localhost:8080//: HTTP 401 Unauthorized.
Jun 04, 2021 9:39:16 AM org.opengrok.indexer.index.Indexer main
 
 Jun 03, 2021 8:56:08 PM org.opengrok.indexer.index.Indexer writeConfigToFile
 INFO: Done...
 Jun 03, 2021 8:56:08 PM org.opengrok.indexer.index.Indexer sendToConfigHost
 INFO: Sending configuration to: http://localhost:8080/sm8250_r_dev_bsui_20201019/
 Jun 03, 2021 8:56:08 PM org.opengrok.indexer.index.Indexer sendToConfigHost
 SEVERE: Failed to send configuration to http://localhost:8080/sm8250_r_dev_bsui_20201019/ (is web application server running with opengrok deployed?)
 java.io.IOException: InboundJaxrsResponse{context=ClientResponse{method=PUT, uri=http://localhost:8080/sm8250_r_dev_bsui_20201019/api/v1/configuration?reindex=true, status=401, reason=Unauthorized}}
    at org.opengrok.indexer.configuration.RuntimeEnvironment.writeConfiguration(RuntimeEnvironment.java:1426)
    at org.opengrok.indexer.index.Indexer.sendToConfigHost(Indexer.java:1149)
    at org.opengrok.indexer.index.Indexer.main(Indexer.java:378)
  
 Jun 03, 2021 8:56:08 PM org.opengrok.indexer.index.Indexer sendToConfigHost
 INFO: Configuration update routine done, check log output for errors.
 Jun 03, 2021 8:56:08 PM org.opengrok.indexer.util.Statistics logIt
 INFO: Indexer finished (took 18:55:16)

02-Jun-2021 12:48:07.826 WARNING [main] org.apache.tomcat.util.scan.StandardJarScanner.processURLs Failed to scan [file:/usr/local/tomcat/webapps/WEB-INF/lib/annotations-19.0.0.jar] from classloader hierarchy
        java.io.IOException: java.lang.reflect.InvocationTargetException
                at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:209)
                at org.apache.tomcat.util.scan.JarFileUrlJar.<init>(JarFileUrlJar.java:65)
                at org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
                at org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:383)
                at org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:318)
                at org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:282)
                at org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:233)
                at org.apache.jasper.servlet.TldScanner.scanJars(TldScanner.java:262)
                at org.apache.jasper.servlet.TldScanner.scan(TldScanner.java:104)
                at org.apache.jasper.servlet.JasperInitializer.onStartup(JasperInitializer.java:83)
                at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5166)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
                at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
                at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:706)
                at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1093)
                at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1810)
                at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
                at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
                at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1013)
                at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:428)
                at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1520)
                at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309)
                at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
                at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
                at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366)
                at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936)
                at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:843)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
                at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
                at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
                at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
                at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
                at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
                at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.StandardService.startInternal(StandardService.java:434)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
                at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
                at org.apache.catalina.startup.Catalina.start(Catalina.java:772)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.base/java.lang.reflect.Method.invoke(Method.java:566)
                at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342)
                at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
        Caused by: java.lang.reflect.InvocationTargetException
                at java.base/jdk.internal.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
                at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
                at org.apache.tomcat.util.compat.Jre9Compat.jarFileNewInstance(Jre9Compat.java:206)
                ... 49 more
        Caused by: java.nio.file.NoSuchFileException: /usr/local/tomcat/webapps/sm8250_r_dev_bsui_20201019/WEB-INF/lib/annotations-19.0.0.jar
                at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
                at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
                at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
                at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
                at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
                at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
                at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
                at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1239)
                at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:732)
                at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:849)
                at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:247)
                at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:177)
                at java.base/java.util.jar.JarFile.<init>(JarFile.java:348)
                ... 53 more
 
https://github.com/oracle/opengrok/issues/3390
原因是 一个依赖了 annotations-19.0.0.jar， 另外一个却依赖 annotations-20.1.0.jar