суббота, 22 сентября 2018 г.

Jenkins slave стал запускаться дольше 5 минут на Raspberry PI 3

После обновления плагинов Jenkins перестал запускаться агент на Raspberry PI 3. Попробовал увеличить таймаут до 5 минут, но это не помогло

[09/22/18 19:23:36] [SSH] Checking java version of java
[09/22/18 19:23:37] [SSH] java -version returned 1.8.0_40-internal.
[09/22/18 19:23:37] [SSH] Starting sftp client.
[09/22/18 19:23:37] [SSH] Copying latest remoting.jar...
[09/22/18 19:23:37] [SSH] Copied 776,265 bytes.
Expanded the channel window size to 4MB
[09/22/18 19:23:37] [SSH] Starting agent process: cd "/home/jenkins" && java  -jar remoting.jar -workDir /home/jenkins
Sep 22, 2018 7:23:53 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/remoting as a remoting work directory
Both error and output logs will be printed to /home/jenkins/remoting
ERROR: null
java.util.concurrent.CancellationException
 at java.util.concurrent.FutureTask.report(FutureTask.java:121)
 at java.util.concurrent.FutureTask.get(FutureTask.java:192)
 at hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:904)
 at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
 at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
 at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
[09/22/18 19:28:36] Launch failed - cleaning up connection
Slave JVM has not reported exit code. Is it still running?
[09/22/18 19:28:36] [SSH] Connection closed.

Зашел по SSH и прицепился к процессу java через strace

$ sudo strace -fF -p 10245
...
[pid 10245] futex(0x76380044, FUTEX_WAIT_BITSET_PRIVATE, 1, {4888, 501190012}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 10245] futex(0x76380028, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 501776999}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 502077571}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 502381685}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 502568039}) = 0
[pid 10245] futex(0x76380044, FUTEX_WAIT_BITSET_PRIVATE, 1, {4888, 552568039}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 10245] futex(0x76380028, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 553154922}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 553459713}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 553807785}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 553995337}) = 0
[pid 10245] futex(0x76380044, FUTEX_WAIT_BITSET_PRIVATE, 1, {4888, 603995337}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 10245] futex(0x76380028, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 604585865}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 604886125}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 605186854}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 605490238}) = 0
[pid 10245] futex(0x76380044, FUTEX_WAIT_BITSET_PRIVATE, 1, {4888, 655490238}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 10245] futex(0x76380028, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 656076704}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 656386131}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 656688214}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 656992067}) = 0
[pid 10245] futex(0x76380044, FUTEX_WAIT_BITSET_PRIVATE, 1, {4888, 706992067}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 10245] futex(0x76380028, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 707577075}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 707885199}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 708183740}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 708487542}) = 0
[pid 10245] futex(0x76380044, FUTEX_WAIT_BITSET_PRIVATE, 1, {4888, 758487542}, ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 10245] futex(0x76380028, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 759065257}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 759258903}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 759556402}) = 0
[pid 10245] clock_gettime(CLOCK_MONOTONIC, {4888, 759863277}) = 0

Похоже идет ожидание какого-то события, но чего именно мне не понятно. При этом полностью нагружено одно из ядер.

Попробовал запустить агент из консоли, чтобы посмотреть сколько на самом деле занимает его запуск

$ cat | java -jar agent.jar -workDir /tmp/jenkins 
Sep 22, 2018 7:55:55 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /tmp/jenkins/remoting as a remoting work directory
Both error and output logs will be printed to /tmp/jenkins/remoting
<===[JENKINS REMOTING CAPACITY]===>

Примерно через 15 минут появился маркер готовности от агента. Это слишком много, т.к. раньше агент запускался за минуту-две максимум. Пока выставил таймаут на 30 минут и агент подключился

[09/22/18 20:31:45] [SSH] Checking java version of java
[09/22/18 20:31:45] [SSH] java -version returned 1.8.0_40-internal.
[09/22/18 20:31:45] [SSH] Starting sftp client.
[09/22/18 20:31:45] [SSH] Copying latest remoting.jar...
[09/22/18 20:31:45] [SSH] Copied 776,265 bytes.
Expanded the channel window size to 4MB
[09/22/18 20:31:45] [SSH] Starting agent process: cd "/home/jenkins" && java  -jar remoting.jar -workDir /home/jenkins
Sep 22, 2018 8:32:01 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/remoting as a remoting work directory
Both error and output logs will be printed to /home/jenkins/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 3.25
This is a Unix agent
Evacuated stdout
Agent successfully connected and online

Комментариев нет:

Отправить комментарий