Inodes not allowing UCS upgrade

I have been running SuiteCRM for a couple of years as a proof-of-concept for our sales team. It has recently seen much more usage and starting to look like something that is worth purchasing the paid version of. However, last week users were reporting to me that when they tried to login, they would get something like ‘login session expired’.

I tried to update UCS from UCS 4.4-8 to UCS 5.0-6 via the management page, but nothing would happen. So when I SSH into the server and tried via console, I would get errors saying not enough space to create temp files.

it is on a VM running Ubuntu, on a 200GB vdisk. So I ran df -h and got

Filesystem               Size  Used Avail Use% Mounted on
udev                      10M     0   10M   0% /dev
tmpfs                    508M   49M  459M  10% /run
/dev/mapper/vg_ucs-root   46G   18G   26G  41% /
tmpfs                    1.3G     0  1.3G   0% /dev/shm
tmpfs                    5.0M     0  5.0M   0% /run/lock
tmpfs                    1.3G     0  1.3G   0% /sys/fs/cgroup
/dev/sda1                472M  265M  183M  60% /boot
overlay                   46G   18G   26G  41% /var/lib/docker/overlay2/641e802f8580d80a960266a9689eae62e94dc44b5462a0f3874bea44af2518be/merged
overlay                   46G   18G   26G  41% /var/lib/docker/overlay2/5f996a9a311ebb850849438229adf79f4a44bbcdf60f2b745ae39cb31cb08522/merged
shm                       64M     0   64M   0% /var/lib/docker/containers/eb43cefbbf70548ab3871f218473423c3a7644dac97bfb34b7e318e1649c99b5/mounts/shm
shm                       64M     0   64M   0% /var/lib/docker/containers/8646d893ec4a363d1ff1c57da229b2dd2aeba089f0ecbccc131bfec55ea28a3c/mounts/shm

then I ran df -i and noticed

Filesystem               Inodes   IUsed  IFree IUse% Mounted on
udev                     318935     346 318589    1% /dev
tmpfs                    324766    1582 323184    1% /run
/dev/mapper/vg_ucs-root 6230112 6230112      0  100% /
tmpfs                    324766       1 324765    1% /dev/shm
tmpfs                    324766       5 324761    1% /run/lock
tmpfs                    324766      15 324751    1% /sys/fs/cgroup
/dev/sda1                124928     362 124566    1% /boot
overlay                 6230112 6230112      0  100% /var/lib/docker/overlay2/641e802f8580d80a960266a9689eae62e94dc44b5462a0f3874bea44af2518be/merged
overlay                 6230112 6230112      0  100% /var/lib/docker/overlay2/5f996a9a311ebb850849438229adf79f4a44bbcdf60f2b745ae39cb31cb08522/merged
shm                      324766       1 324765    1% /var/lib/docker/containers/eb43cefbbf70548ab3871f218473423c3a7644dac97bfb34b7e318e1649c99b5/mounts/shm
shm                      324766       2 324764    1% /var/lib/docker/containers/8646d893ec4a363d1ff1c57da229b2dd2aeba089f0ecbccc131bfec55ea28a3c/mounts/shm

Showing the two docker overlay2 hogging all of the index nodes. I am not well versed on SuiteCRM and I have done a lot of research looking at how to clean these docker containers, but I have been unsuccessful. There is a lot of data and custom modules that I would really like to be able to save. I am hoping someone may be able to point me in the right direction.

Once corrected, would it be beneficial to upgrade UCS? What proactive measures can I take to avoid this in the future?

Look at the session directories in the containers.

Some misconfigured or buggy PHPs leave endless session files without purging.

See your php.ini, session.save_path configuration.

Thank you!

I pulled up my php.ini located in /etc/php/7.0/apache2/

However, any line that has the sessions.save_path config is commented out. This CRM was the default SuiteCRM Ubuntu VM image I pulled off of the site. I can’t say I modified much of it from the default.

I would assume the default location would be a tmp folder of some kind, I checked /var/lib/php, but I only have a modules folder in that directory, no sessions folder.

Trying to prune docker doesn’t reclaim any space. Which an abundance of PHP sessions makes sense. I am just at a loss as to how to locate and/or remove these sessions. Alos, if there is a way to prevent this from happening again.

The default location for PHP sessions on the bitnami container is:
session.save_path /opt/bitnami/php/var/run/session

The tip from Chris should solve your problem, but anyway let me just say the generic answer is to get the effective session.save_path value from phpinfo

For CLI, that is as simple as typing

php -i | grep save_path

For the web server, you can go in Admin / Diagnostics and get a phpinfo output. Or create a simple phpinfo file in your server (many tutorials online will teach you this).

Another option would be to go inside the container and run a series of df -i, drilling down to the exact directory.

I am sorry if my inexperience is showing. While I have used Docker a lot in the past, I have never really done much other than run pre-built containers and never really modified or played around with the inner-workings.
Locating the PHP sessions has be a bit stumped.

root@crm:~# php -i | grep save_path
session.save_path => /var/lib/php/sessions => /var/lib/php/sessions
root@crm:~# cd /var/lib/php/sessions
-bash: cd: /var/lib/php/sessions: No such file or directory

df -i still shows the 2 docker overlays at 100% IUse.

overlay                 6230112 6230112      0  100% /var/lib/docker/overlay2/641e802f8580d80a960266a9689eae62e94dc44b5462a0f3874bea44af2518be/merged
overlay                 6230112 6230112      0  100% /var/lib/docker/overlay2/5f996a9a311ebb850849438229adf79f4a44bbcdf60f2b745ae39cb31cb08522/merged

running ls -lai in both directory shows this, I just don’t know if/what is ok do remove.

root@crm:/var/lib/docker/overlay2/5f996a9a311ebb850849438229adf79f4a44bbcdf60f2b745ae39cb31cb08522/merged# ls -lai
total 1152
   27848 drwxr-xr-x 1 root root   4096 Feb  7 21:10 .
 5783373 drwx------ 5 root root   4096 Feb 22 10:14 ..
   29831 drwxr-xr-x 1 root root   4096 Nov  6  2019 bin
  259772 drwxr-xr-x 1 root root   4096 Oct  7  2019 boot
 5758290 -rw------- 1 root root 651264 Feb  7 21:10 core
   28768 drwxr-xr-x 1 root root   4096 Nov  5  2019 dev
 5783411 -rwxr-xr-x 1 root root      0 Nov  5  2019 .dockerenv
   28722 drwxr-xr-x 1 root root 708608 Feb 22 10:15 etc
27684174 drwxr-xr-x 2 root root   4096 Nov 19  2017 home
 5782887 lrwxrwxrwx 1 root root     30 Oct  7  2019 initrd.img.install -> boot/initrd.img-4.9.0-11-amd64
   29835 drwxr-xr-x 1 root root   4096 Nov  6  2019 lib
   29833 drwxr-xr-x 1 root root   4096 Oct  7  2019 lib64
  242738 drwxr-xr-x 2 root root   4096 Feb 12  2018 media
  258075 drwxr-xr-x 2 root root   4096 Feb 12  2018 mnt
  242735 drwxr-xr-x 2 root root   4096 Feb 12  2018 opt
   28767 drwxr-xr-x 2 root root   4096 Nov 19  2017 proc
   29986 drwx------ 1 root root   4096 Nov  5  2019 root
   28775 drwxr-xr-x 1 root root   4096 Oct  7  2019 run
13213355 drwxr-xr-x 1 root root   4096 Nov  6  2019 sbin
27684173 drwxr-xr-x 2 root root   4096 Feb 12  2018 srv
   28771 drwxr-xr-x 1 root root   4096 Nov  5  2019 sys
   30509 drwxrwxrwt 1 root root   4096 Feb 22 11:26 tmp
 5774053 -rw-r--r-- 1 root root    127 Nov 12  2018 .univention-environment
   29898 drwxr-xr-x 1 root root   4096 Feb 21  2019 usr
   28777 drwxr-xr-x 1 root root   4096 Nov  5  2019 var
 5782890 lrwxrwxrwx 1 root root     27 Oct  7  2019 vmlinuz.install -> boot/vmlinuz-4.9.0-11-amd64

and

root@crm:/var/lib/docker/overlay2/641e802f8580d80a960266a9689eae62e94dc44b5462a0f3874bea44af2518be/merged# ls -lai
total 4220
   27846 drwxr-xr-x 1 root root    4096 Feb 28 21:47 .
  800011 drwx------ 5 root root    4096 Feb 22 10:14 ..
   29878 dr-xr-xr-x 1 root root    4096 Jul 31  2019 bin
  251678 drwxr-xr-x 3 root root    4096 Jul 31  2019 boot
  800069 -rw------- 1 root root 4485120 Feb 28 21:47 core
   28814 drwxr-xr-x 1 root root    4096 Dec  5  2019 dev
  797036 -rwxr-xr-x 1 root root    6190 Apr  4  2019 docker-entrypoint.sh
  800022 -rwxr-xr-x 1 root root       0 Dec  5  2019 .dockerenv
   28262 drwxr-xr-x 1 root root    4096 Dec  5  2019 etc
   28821 drwxr-xr-x 4 root bin     4096 Jul 31  2019 fwxserver
  251629 drwxr-xr-x 1 root root    4096 Jul 31  2019 home
  251631 dr-xr-xr-x 1 root root    4096 Jul 31  2019 lib
   29880 dr-xr-xr-x 1 root root    4096 Jul 31  2019 lib64
16093427 drwx------ 2 root root    4096 Jun  2  2016 lost+found
16093428 drwxr-xr-x 2 root root    4096 Sep 23  2011 media
16093429 drwxr-xr-x 2 root root    4096 Sep 23  2011 mnt
12922468 drwxr-xr-x 2 root root    4096 Sep 23  2011 opt
   28835 drwxr-xr-x 1 root bin     4096 Jul 31  2019 private
   28813 drwxr-xr-x 2 root root    4096 Jun  2  2016 proc
   29051 dr-xr-x--- 2 root root    4096 Jun  2  2016 root
   28819 drwxr-xr-x 2 root root    4096 Dec  5  2019 run
   30001 dr-xr-xr-x 1 root root    4096 Jul 31  2019 sbin
   30048 drwxr-xr-x 2 root root    4096 Jun  2  2016 selinux
16093430 drwxr-xr-x 2 root root    4096 Sep 23  2011 srv
   28817 drwxr-xr-x 1 root root    4096 Dec  5  2019 sys
   30173 drwxrwxrwt 1 root root    4096 Feb 22 11:26 tmp
   28830 drwxr-xr-x 1 root root    4096 Jun  2  2016 usr
   28860 drwxr-xr-x 1 root root    4096 Dec  5  2019 var

Both directories have tmp folders that seem to be quite large. I just don’t know if I can/should clean them out.

root@crm:/var/lib/docker/overlay2/641e802f8580d80a960266a9689eae62e94dc44b5462a0f3874bea44af2518be/merged/tmp# ls -lai
total 1560564
   30173 drwxrwxrwt 1 root root      4096 Feb 22 11:26 .
   27846 drwxr-xr-x 1 root root      4096 Feb 28 21:47 ..
 1844647 -rw------- 1 root root     19116 Jun  2  2016 anaconda-post.log
  792687 -rw------- 1 root root       340 Feb 22 10:15 apache-stderr---supervisor-t3NUXR.log
  792686 -rw------- 1 root root         0 Feb 22 10:15 apache-stdout---supervisor-1oUQay.log
16185559 drwxr-xr-x 1 root root      4096 Jul 31  2019 filewave
  803098 -rwxr-xr-x 1 root root       150 Mar 22  2019 filewave-configure
  792840 -rw------- 1 root root   6660746 Feb 28 21:47 fwldap-stderr---supervisor-MFNYCD.log
  792838 -rw------- 1 root root         0 Feb 22 10:15 fwldap-stdout---supervisor-1oNw7x.log
  792747 -rw------- 1 root root         0 Feb 22 10:15 fwxserver_a-stderr---supervisor-_AC9xP.log
  792746 -rw------- 1 root root         0 Feb 22 10:15 fwxserver_a-stdout---supervisor-pNdJ12.log
  894116 -rw------- 1 root root  19930324 Mar 18  2020 fwxserver.h24207
  893858 -rw------- 1 root root  27190692 Mar  7  2020 fwxserver.h24683
  894112 -rw------- 1 root root  89083854 Mar 12  2020 fwxserver.h25060
 5511886 -rw------- 1 root root 219922431 Sep  9 10:38 fwxserver.h31930
  894113 -rw------- 1 root root  17324214 Mar 13  2020 fwxserver.h49226
  894117 -rw------- 1 root root  14745634 Mar 23  2020 fwxserver.h61655
 1050615 -rw------- 1 root root  69324032 Jul 31  2019 fwxserver.kn1119
  889555 -rw------- 1 root root  25904228 Mar  5  2020 fwxserver.kn3436
  894114 -rw------- 1 root root 142861118 Mar 14  2020 fwxserver.kn7790
  894110 -rw------- 1 root root  97014544 Mar  9  2020 fwxserver.kn7842
  792819 -rw------- 1 root root         0 Feb 22 10:15 fwxserver_s-stderr---supervisor-V4vPo6.log
  792815 -rw------- 1 root root         0 Feb 22 10:15 fwxserver_s-stdout---supervisor-XLEVRS.log
 1037144 -rw------- 1 root root 370245632 Feb 14  2023 fwxserver.TwZ455
  894118 -rw------- 1 root root 177767747 Mar 26  2020 fwxserver.TwZ489
  894111 -rw------- 1 root root  54859961 Mar  9  2020 fwxserver.TwZ493
  811871 -rw------- 1 root root  11529300 Dec  5  2019 fwxserver.TwZ537
  894115 -rw------- 1 root root 253512665 Mar 16  2020 fwxserver.UwZ493
  792685 -rw------- 1 root root       649 Feb 22 10:15 postgres-stderr---supervisor-RpAYE1.log
  792684 -rw------- 1 root root         0 Feb 22 10:15 postgres-stdout---supervisor-sIgFsa.log
 6016667 -rw-r----- 1 root root         0 Feb 22 11:25 qipc_sharedmemory_FWRuntimeControl2f384e01b0abf1251baa0003ea43f8a9d0c3e693
 4114654 srwxrwxrwx 1 root root         0 Feb 19 09:39 SAK1
  792688 -rw------- 1 root root     31934 Feb 28 10:45 scheduler-stdout---supervisor-hFfZdZ.log
  792869 -rw------- 1 root root        78 Feb 22 10:15 update_controller-stderr---supervisor-RWbFGu.log
  792841 -rw------- 1 root root         0 Feb 22 10:15 update_controller-stdout---supervisor-5jnYUA.log
 1844649 -rw------- 1 root root         0 Jun  2  2016 yum.log

root@crm:/var/lib/docker/overlay2/5f996a9a311ebb850849438229adf79f4a44bbcdf60f2b                                                                                                             745ae39cb31cb08522/merged/tmp# ls -lai
total 28
   30509 drwxrwxrwt 1 root root 4096 Feb 22 11:26 .
   27848 drwxr-xr-x 1 root root 4096 Feb  7 21:10 ..
13080400 drwxrwxrwt 2 root root 4096 Feb 22 10:15 .font-unix
13080398 drwxrwxrwt 2 root root 4096 Feb 22 10:15 .ICE-unix
13080401 drwxrwxrwt 2 root root 4096 Feb 22 10:15 .Test-unix
13080397 drwxrwxrwt 2 root root 4096 Feb 22 10:15 .X11-unix
13080399 drwxrwxrwt 2 root root 4096 Feb 22 10:15 .XIM-unix

Maybe it’s easier to search with this command:

du --inodes --max-depth=1

You can also use a depth of 2 if you find it heplful

The idea is for you to drill down. See which directory has the most inodes, go into it, see what is underneath that in terms of inodes. Eventually you will get there.

About the value defined in php.ini, maybe you got that from outside the container, not from the inside?

Thank you to everyones help here, I was able to clean up the sessions folder. I didn’t realize there was a similar root directory within the docker overlay, THAT was where my PHP sessions directory was. After cleaning that up,

df -i now looks like this,

root@crm:/var/lib/docker/overlay2/5f996a9a311ebb850849438229adf79f4a44bbcdf60f2b745ae39cb31cb08522/merged# df -i
Filesystem               Inodes   IUsed   IFree IUse% Mounted on
udev                     318935     346  318589    1% /dev
tmpfs                    324766    1570  323196    1% /run
/dev/mapper/vg_ucs-root 6230112 3003464 3226648   49% /
tmpfs                    324766       1  324765    1% /dev/shm
tmpfs                    324766       6  324760    1% /run/lock
tmpfs                    324766      15  324751    1% /sys/fs/cgroup
/dev/sda1                124928     362  124566    1% /boot
overlay                 6230112 3003464 3226648   49% /var/lib/docker/overlay2/5f996a9a311ebb850849438229adf79f4a44bbcdf60f2b745ae39cb31cb08522/merged
overlay                 6230112 3003464 3226648   49% /var/lib/docker/overlay2/641e802f8580d80a960266a9689eae62e94dc44b5462a0f3874bea44af2518be/merged
shm                      324766       2  324764    1% /var/lib/docker/containers/8646d893ec4a363d1ff1c57da229b2dd2aeba089f0ecbccc131bfec55ea28a3c/mounts/shm
shm                      324766       1  324765    1% /var/lib/docker/containers/eb43cefbbf70548ab3871f218473423c3a7644dac97bfb34b7e318e1649c99b5/mounts/shm

However, for some reason, when someone tries to login, we still get

Which I don’t quite understand why, I can login to Univention Portal no problem, is that because SuiteCRM is running in a Docker container?

Needless to say, I was pretty bummed out when I came in this morning seeing that I had successfully freed the majority of inodes, only to realize users are still unable to login. Which is odd because when I went in and deleted a bunch of log files, freeing up ~1% of inodes, I was able to login once, another user was able to login once, then it was back to the session has expired.

Even though the inodes are free, I am still showing an insane amount of sessions in the sessions directory still.

# sudo find var/lib/php/sessions/. -type f|wc -l      
2697427

It doesn’t look like you cleaned up the directory yet…

I get the feeling that you’re working blind-folded :man_with_probing_cane:

Now that you found out the container, and the directory, can’t you just go in there with normal cd command and check what’s there, and delete what is not needed?

Start a bash session within the container, and cd into the sessions directory.