k8s Pod unable to create new native thread

java.lang.OutOfMemoryError: unable to create new native thread

Exception in thread "ActiveMQ Task-551" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

ulimit -a

core file size (blocks, -c) 0

data seg size (kbytes, -d) unlimited

scheduling priority (-e) 0

file size (blocks, -f) unlimited

pending signals (-i) 1029315

max locked memory (kbytes, -l) 64

max memory size (kbytes, -m) unlimited

open files (-n) 65535

pipe size (512 bytes, -p) 8

POSIX message queues (bytes, -q) 819200

real-time priority (-r) 0

stack size (kbytes, -s) 8192

cpu time (seconds, -t) unlimited

max user processes (-u) 40960

virtual memory (kbytes, -v) unlimited

file locks (-x) unlimited

Understanding /etc/security/limits.conf file

https://www.thegeekdiary.com/understanding-etc-security-limits-conf-file-to-set-ulimit/

[domain] can be:

• an user name
• a group name, with @group syntax
• the wildcard *, for default entry
• the wildcard %, can be also used with %group syntax, for maxlogin limit

[type] can have the two values:

• “soft” for enforcing the soft limits
• “hard” for enforcing hard limits

[item] can be one of the following:

• core – limits the core file size (KB)
• data – max data size (KB)
• fsize – maximum filesize (KB)
• memlock – max locked-in-memory address space (KB)
• nofile – max number of open files
• rss – max resident set size (KB)
• stack – max stack size (KB)
• cpu – max CPU time (MIN)
• nproc – max number of processes
• as – address space limit (KB)
• maxlogins – max number of logins for this user
• maxsyslogins – max number of logins on the system
• priority – the priority to run user process with
• locks – max number of file locks the user can hold
• sigpending – max number of pending signals
• msgqueue – max memory used by POSIX message queues (bytes)
• nice – max nice priority allowed to raise to values: [-20, 19]
• rtprio – max realtime priority

man limits.conf

修改docker 容器pids-limit限制

/etc/sysconfig/docker

# Modify these options if you want to change the way the docker daemon runs

OPTIONS=' --selinux-enabled --selinux-enabled --insecure-registry 10.161.0.0/16 --log-driver json-file --log-opt max-size=50M --log-opt max-file=3 --bip=10.41.199.1/24 --insecure-registry harbor.cloud.bz --add-registry harbor.cloud.bz --signature-verification=False --default-pids-limit=40960'

if [ -z "${DOCKER_CERT_PATH}" ]; then

DOCKER_CERT_PATH=/etc/docker

fi

# Do not add registries in this file anymore. Use /etc/containers/registries.conf

# instead. For more information reference the registries.conf(5) man page.

# Location used for temporary files, such as those created by

# docker load and build operations. Default is /var/lib/docker/tmp

# Can be overriden by setting the following environment variable.

# DOCKER_TMPDIR=/var/tmp

# Controls the /etc/cron.daily/docker-logrotate cron job status.

# To disable, uncomment the line below.

# LOGROTATE=false

# docker-latest daemon can be used by starting the docker-latest unitfile.

# To use docker-latest client, uncomment below lines

#DOCKERBINARY=/usr/bin/docker-latest

#DOCKERDBINARY=/usr/bin/dockerd-latest

#DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest

#DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest

ADD_REGISTRY='--add-registry registry.redhat.io'

How to set nproc (Hard and Soft) Values in CentOS / RHEL 5,6,7

https://www.thegeekdiary.com/how-to-set-nproc-hard-and-soft-values-in-centos-rhel-567/

How to Increase Number of Open Files Limit in Linux

https://www.tecmint.com/increase-set-open-file-limits-in-linux/

Find Linux Open File Limit

The value is stored in:

# cat /proc/sys/fs/file-max

818354

Check Hard Limit in Linux

# ulimit -Hn

4096

Check Soft Limits in Linux

# ulimit -Sn

1024

To see the hard and soft values for different users, you can simply switch user with “su” to the user which limits you want to check.

For example:

# su marin
$ ulimit -Sn

1024

$ ulimit -Hn

4096

How to Check System wide File Descriptors Limits in Linux

If you are running a server, some of your applications may require higher limits for opened file descriptors. A good example for such are MySQL/MariaDB services or Apache web server.

You can increase the limit of opened files in Linux by editing the kernel directive fs.file-max. For that purpose, you can use the sysctl utility.

Sysctl is used to configure kernel parameters at runtime.

For example, to increase open file limit to 500000, you can use the following command as root:

# sysctl -w fs.file-max=500000

You can check the current value for opened files with the following command:

$ cat /proc/sys/fs/file-max

With the above command the changes you have made will only remain active until the next reboot. If you wish to apply them permanently, you will have to edit the following file:

# vi /etc/sysctl.conf

Add the following line:

fs.file-max=500000

Of course, you can change the number per your needs. To verify the changes again use:

# cat /proc/sys/fs/file-max

Users will need to logout and login again for the changes to take effect. If you want to apply the limit immediately, you can use the following command:

# sysctl -p

Set User Level Open File limits in Linux

The above examples, showed how to set global limits, but you may want to apply limits per user basis. For that purpose, as user root, you will need to edit the following file:

# vi /etc/security/limits.conf

If you are a Linux administrator, I suggest you that you become very familiar with that file and what you can do to it. Read all of the comments in it as it provides great flexibility in terms of managing system resources by limiting users/groups on different levels.

The lines that you should add take the following parameters:

<domain>        <type>  <item>  <value>

Here is an example of setting a soft and hard limits for user marin:

## Example hard limit for max opened files
marin        hard nofile 4096
## Example soft limit for max opened files
marin        soft nofile 1024