Load Issues
"Thats the biggest load I have ever seen"
"Thats what she said Jim.."
Got a funny Load Meme. Contact me to share it!
Logs can tell you a lot about a site and is traffic. Lets touch on some ways to troubleshoot with Log Files.
Sorting / Viewing Apache Access Logs Traffic
To Start we are going to CD to the access logs folder and list the available sites (log files)
For cPanel this location is
You should return a similar result (You may have more domains)
Show Amount of site requests Per Day (hits to the log file).
You can also save the output to a file
this should return
Next, You can break this down per hour
Which should return an Hourly Report
Next, Per minute
Which should return
Excessive Site / Server Requests
Tracking a Malicious or potentailly dangerous user
First we will start by navigating to the access logs and listing the Specific Logs available
Which should return a similar result
Next we will use the awk command to print the 1st column of the Apache log (IP address),.
Then pipe | that to the sort -n command so all of the IPs are listed numerically,
Then a further pipe to the uniq -c command count percisely how many times each IP hits the server/site.
Finally pipe all that back to the sort -n command which then lists the IPs based on how many requests they made
This should return a List of Hits per IP
Now the above shows us the IP 11.22.11.22 is a 'heavy hitter'. Next we want to see what they are acessing or doing
to return
In the results we can see that this user is hitting the wp-login.php Script with a GET then a POST..
A GET then POST indicates this user arrived at the page(GET), then tried to log in (POST) 727 times
At this point you can take any Directed action on the Malicious IP to prevent further issues
Apache Log Request Types
Ouu this is one of my Favorites!
As usual with any Logs cd to the Log Location and list the available log files
Which should return your available access logs
View Request Types: HEAD, POST, GET
GET means a visitor is simply requesting a resource such as a HTML page or image,
HEAD is typically a web-browser or bot checking to see if the file requested has been updated since it was last accessed.
POST means a visitor has filled out information in a form and is POSTing it to the server much like you would see from a login attempt.
which should return a similar result
View Most Requested base URLs
Which should return something like
View Most Requested Unique URLs
great way to check if a specific URL is being hit
which returns
View specific response codes - 301, 403, 500, etc..
which returns
Tracking User Agents
As usual with any Logs cd to the Log Location and list the available log files
Which should return your available access logs
Searching for a Specific User Agent
returns (example)
With the above list you can see the AhrefsBot is the biggest User agent. You can then take action VIA robots.txt file to block this specific SearchEngineBot
Troubleshooting Server Usage Spikes
what to do if your servers load average is spiking
the sar command
Verify sar Version
Output CPU Statistics X time, X Seconds apart - Example 3 Times, 1 Second apart
Report memory usages.Example 3 Times, 1 Second apart
Look for “kbmemfree” and “kbmemused”
Report I/O Usages. Example 3 Times, 1 Second apart
-
- tps – Transactions per second (this includes both read and write)
- rtps – Read transactions per second
- wtps – Write transactions per second
- bread/s – Bytes read per second
- bwrtn/s – Bytes written per second
As usual with any Logs cd to the Log Location and list the available log files
Which should return your available access logs
Troubleshooting Load Averages
Check server load average over the last minute - 12 Times, 5 Seconds Apart
Looking at the reports we see a spike just as we started the scan so we are going to use this in the example: 12:12 AM
We can see this got as high as 1.80
This however is not high enough for troubleshooting. I switched to a more busy server for this example and was still unable to achieve a high load. For this example we will continue stating the above is a high load: 1.80
Commonly Sites being accessed run PHP Code or other server sided code which can also cause usage spikes. Be sure to also check your Apache Access Logs to see what was goign on around that time.
With the below we will track the hits prior and up to the Load Spike from 12:11AM - 12:13AM
Above we see 622 Request over this time period. We can further break down the requests per minute with the following
We can see above the Hits to the site are Double in comparison to a few moments before the scan.
Next we get even Deeper and take a look at what is actually going on with those requests.
Word to the wise:
A server should without issues be able to handle around 100 standard image or HTML Page requests with less resource usage than your average PHP Script.
We can use the below search string to see what Duplicate Requests have been happening
Server Access Logs
Display active IP 'hits' to a server
Netstat Stuff
Display Currently Active established connections
Display Currently Active established connections on Port 80 (standard Apache Port)
Useful in detecting a single flood by allowing you to recognize many connections coming from one IP.
this shows the amount of 'SYNC_REC' which are occurring. This should be ideally Low (less than 5.
On DoS attack incidents or mail bombs, the number can spike.
This value also depends on system, so a high value may be average on another server.
List the specific IPs Involved not just count (wc) active connections
List unique IPs that are sending SYN_REC connection status
Calculate and count the IP Connections to the server
Count and list Active IP Connections Over TCP & UDP Protocol
Count and list ALL IP with an Established Connections instead of all connections
Count, List and show ALL connections to Port 80
Generates a list of IP address preceded by the number of times it hit a site
Generates a list that shows the last 10,000 hits to a site.
Listing top files, folders, and domains
Generates a list of files or directories on your site being called the most.
Generates a list of traffic for all domains listed under a specific user (on a shared server)
This command must be run in your /logs/ directory.
should return
Its My Wiki, for you, for me and for them! Want to contribute, contact me.
2017 Ⓒ
Bitches