The colleague reacted that the system could not log in to the test environment, decisively checked the monitoring, and found that the service port was open, and logged in to the server to check the hard disk usage and found that the directory mounted on a certain disk was full, as shown in the figure below:
The disk is 100% the disk used by the docker application, and it is initially suspected that the inability to write log files prevents the module from providing services to the outside world.
Look for large files and finally locate them under the /var/lib/docker/containers/ subdirectoryA certain json.log file is taking up 30G。
Cleaning Running Container Logs (Symptom Treatment)
In order to allow the container to serve normally immediately. Temporary solution, quickly clean up such files, free up hard disk space, the script is as follows:
If the docker container is running, then after deleting the logs using rm -rf, you will find that disk space is not freed up via df -h. The reason is that on Linux or Unix systems, deleting a file via rm -rf or a file manager will unlink it from the directory structure of the file system. If the file is opened (and there is a process in use), the process will still be able to read the file and disk space will be taken up all the time. The correct pose is cat /dev/null > *-json.log, and of course you can also restart docker after removing it via rm -rf.
Remark:Create a new script in the directory of the free disk, otherwise the script cannot be created normally。
Setting the Docker Container Log Size (Root Cause)
Set the maximum log size of a container service, and add a parameter to set the log size of the container when starting the container, for example:
max-size=500m, which means that the upper limit of the log size of a container is 500M,
max-file=3, which means that a container has three logs, namely id+.json, id+1.json, and id+2.json.
Docker migrated to a new disk
docker-related data exists on the system disk, the system disk is only 50G in size, and the data disk is 100G in size, how to move the existing data of docker to the data disk?
For docker installed using the CentOS system yum method, the default installation directory should be: /var/lib/docker
Docker version < v17.05.0
Because dockerd can specify the image and container storage path through the parameter graph, such as –graph=/var/lib/docker, we only need to modify the configuration file to specify the startup parameters.
Docker's configuration file can set most of the background process parameters, and the storage location in each operating system is inconsistent, in Ubuntu: /etc/default/docker, and in CentOS: /etc/sysconfig/docker.
Docker version >= v17.05.0
Because Docker officially deprecated the graph feature in this release, if you have Docker version >= v17.05.0 installed on your machine, you cannot modify the default installation (storage) directory of Docker by specifying the graph parameter in the /etc/default/docker configuration file.
There are other ways in which the new version of Docker can achieve our goal of modifying the installation (storage) directory: by modifying (new) /etc/docker/daemon.json to specify the value of the data-root parameter.
Docker daemon directory configuration:The hyperlink login is visible.
Migration preparation: Stop all container applications and stop docker services (systemctl stop docker).
Create a new directory in /home/software with the following command:
Migrate the /var/lib/docker directory to the /home/software/docker directory (it is recommended to copy it, and then delete the /var/lib/docker directory file when it is fine), the command is as follows:
When copyingBe sure to add the -a parameter, otherwise some containers will not work properly！！！！！
View all soft connections and file points in the current directory with the following command:
Modify the /etc/docker/daemon.json file with the following command:
Save, then start the docker service with the following command:
Start the container, the tests are all running normally, and finally,Delete the /var/lib/docker folder。
(End)
|
Posted on 2/7/2022 10:21:43 AM
|
|
|
|
Posted on 2/13/2022 4:36:29 PM
|
Landlord|
Posted on 2/24/2022 9:38:26 AM
|
