For the xth time, I’ve hit the MySQL bug #51325 - It state that with relatively big (>5 gigs) innodb buffer_pool (my.cnf::innodb_buffer_pool_size), the LRU invalidation on ‘DROP TABLE’ (or ‘TRUNCATE TABLE’) will lock the full table (even if DROP or TRUNCATE is on a partition). Bug is only present when using the ‘innodb_file_per_table‘, which is kind of sad because I love that configuration setting.
Solutions include disabling the innodb_file_per_table setting & reducing the buffer_pool size…
Along with development for Theatre.IO (server management system) (dev. version available for Les Laboratoires Phoenix enterprise clients), I’ve had to look into geo-diversity for the datacenter where the company got ‘silo’/'reference stack’ (a group of servers & devices which are the same, standardized, everywhere).
After a couple of very funny – they must have been a joke) – quotes from well known providers, look like we’ll be doing a dual deployment (Seattle & Chicago). That should cover north-America quite well. The phase 3, our Europe site, being on hold for another 5 months. This is a major investment for Les LabsPhoenix as our reference stack cost are in the 5 digits realm, but with geo-diversity and our own IPv4 and IPv6 ranges, there is much that can be accomplished.
Stay tuned as I’ll soon be presenting that reference stack (the free software which compose it, and the not-so-free-ones). There is still some things that need to be figured out (constant evolution, redesign), such as if the Mikrotik RB1200 will be powerful enough to hold the main traffic.
Microsoft IIS 5/6 FTP 0Day released
We are aware of an new 0-day exploit that was posted on Milw0rm today.
According the exploit, it was suppose to work on both IIS 5.0 and 6.0, on the FTP module.
Also according it, it affects IIS 6.0 with stack cookie protection.
The latest on this is that HDMoore is porting it to the MetaSploit framework.
We will update this diary with more info as we get it.
source: sans.org
Lets dwell into the technical world that had me captive for a couples hours, last week:
The installation of NGINX & Phusion Passenger should be very straight forward and no cause for a blog post. Except, and the fun start here, if you are compiling both from source, inside Amazon AWS, on a small EC2 instance (32 bits) and your dom0 is 64bits (no way of knowing, but most should) and Amazon gave you a slice of memory over the 4gb of normally addressable slot (small windows of opportunity, but still). How bad can it get ? Well, you will be flooded by the dreaded “4gb seg fixup” error message in your log files and your ruby process will drop to an almost standstill speed.
Your god, Google, will advice you to do stuff (which you should have already tried):
apt-get install libc6-xen
echo ‘hwcap 0 nosegneg’ > /etc/ld.so.conf.d/libc6-xen.conf ; ldconfig
mv /lib/tls /lib/tls.disabled
And this is where you will start to despair, because, of the 26 200 Google results for “4gb seg fixup“, 26 000 are either linking to a post asking the same question or answering with one of those two answers and 200 are for 4gb usb key. Neither of which will help the message to go away from your syslog and bring it back NGINX/Ruby to decent speed.
So. At this point, where you are starting to think about wiping everything and starting back from scratch (which won’t help), try this little procedure. The principle is to remove the passenger gem from your system, reinstall it (which will only download the source), modify the makefile, recompile NGINX (which in turn automatically compile the Phusion Passenger module) and take a beer while your system serve ruby pages without (systems) errors.
#> gem uninstall passenger
#> gem install passenger
We have a valid passenger gem source code in /var/lib/gems/1.8/gems/passenger-2.2.4 - version can vary and location is valid for Ubuntu/Debian, but could change on others distro. We will be modifying the optimization flags given to the compiler. Since Phusion Passenger does not accept command line argument and variables declarations, we have no other choices than to modify the rake file pre-compilation.
#> sed ‘s/EXTRA_CXXFLAGS = “-Wall #{OPTIMIZATION_FLAGS}”/EXTRA_CXXFLAGS = “-Wall -mno-tls-direct-seg-refs #{OPTIMIZATION_FLAGS}”/g’ /tmp/rakefile
#> mv /tmp/rakefile /var/lib/gems/1.8/gems/passenger-2.2.4/Rakefile
This being done, we will start an NGINX compilation process which will, in turn, start passenger-2.2.4 compilation. Using the -mno-tls-direct-seg-refs will allows us to work arround the 4gb seg fixup error.
#> CFLAGS=”-mno-tls-direct-seg-refs” CXXFLAGS=”-mno-tls-direct-seg-refs” ./configure –prefix=’/usr/local/nginx-0.7.61′ –add-module=’/var/lib/gems/1.8/gems/passenger-2.2.4/ext/nginx’ –with-http_ssl_module –with-http_stub_status_module
There you go.
You gotta love those SUN’s Fishwork Engineer…
In the following video, Brendan Gregg plot I/O operations per seconds by latency and number of I/O ops over a defined time threshold while some “vibration” are introduced. Bottom line : Do not yell at your computer. You are not helping.
Youtube video: Bredan Gregg & hd vibrations