Here’s a time laps video I made a few months ago, a couple of hours in the studio.
Studio.
dentarg and ludde were going for a flight so I tagged along with my camera, trying to take some pictures. It’s amazing how different everything looks from 1500 feet up in the air.
Pretty hard to take photos though, seemed like none of my lenses were good in the plane, probably just me sucking too much. My 17-55 mm lens was not wide enough to get the interior shot I wanted, and the 100 – 300 mm was either too narrow, or the windows were too dirty to get good shots of the ground. I should have brought the 35 – 105 mm, but since I woke up 5 minutes before they came to pick me up, I didn’t think far enough.
WIntellect from BSDNexus was interviewed by BSDtalk last night, go listen to it here!
(It’s strange to hear “jage” pronounced in english.)
Igårkväll gjorde vi en förändring i hur våran webbhosting fungerar, hur man hanterar virtuella hostnamn och hur vi administrerar det hela.
Tidigare hade vi stora Apache-konfigurationer där alla virtuella hostnamn (vhost) stod specificerade, och behövdes uppdateras manuellt. Varje vhost krävde 6 rader text för att fungera, vilket var tidskrävande att hantera.
Nu har vi istället gjort ett halvautomatiskt system, där Apaches konfiguration byggs upp automatiskt med hjälp av ett script som letar reda på alla vhosts. Detta kräver dock en strikt katalogstruktur, vilket har mer än denna fördel, för att fungera.
Varje användare har en katalog under /var/www/users/ där användaren lägger upp sitt material och bestämmer hur vhostarna skall “peka”. I katalogen ser det ut såhär:
$ ls -l /var/www/users/jage
drwxr-xr-x 6 jage www 512 Dec 30 20:30 htdocs/
drwxr-xr-x 5 jage www 512 Jan 4 01:46 vhosts/
Varje användare har en subdomän till starkast.net, t.ex jage.starkast.net som alltid pekar till htdocs-katalogen. Vanligast är att man bara använder detta, utan egna domäner. Om man dock vill ha egna domäner behöver man titta på vhosts-katalogen, där användarna skapar kataloger som bestämmer hur domänerna mappas på servern.
$ ls -l /var/www/users/jage/vhosts
lrwxr-xr-x 1 jage jage 9 Jan 4 01:46 blog.jage.se@ -> ../htdocs
drwxr-xr-x 6 jage jage 512 Dec 31 13:05 imum.net/
lrwxr-xr-x 1 jage jage 9 Jan 4 01:45 jage.se@ -> ../htdocs
Här ser vi kataloger och symlänkar som har exakt samma namn som domänen i fråga. Symlänkar använder man om man inte vill ha en separat katalog där innehållet ligger, utan kanske vill ha samma innehåll som via en annan domän. Jag har till exempel skapat en symlänk från blog.jage.se tilll ../htdocs, jage.se går alltså till samma katalog på servern som jage.starkast.net. Däremot imum.net vill jag ha separerat, därför är det en helt vanlig katalog.
Med hjälp av det här systemet kan man ändra om sina vhosts utan att någon systemadministratör behöver hjälpa en. Användaren blir mer självständig.
Woho, I’m almost done with backporting my old design I had with the Typo installation. Did some tweaking with the font sizing, taking some hints from Dan Cederholm’s Bulletproof Web Design I read earlier this year. Pretty good book by the way, even though I’m really not in any HTML/CSS-mood anymore.
I really like this “non-design” though. :-)
I don’t care if it’s fake or not, it’s so hilarious!
Two great photographers.
Zena Holloway
Tim Flach
I’ve been using Typo for half a year now, posting occasionally about some things that interest me, but I’m mostly too lazy to write about the cool stuff I encounter or experience. I’ve just migrated from typo to Mephisto, and with this, hoping to improve my “blogging”. I want to post more about things that inspire me, and about my life. Two different “sections”, would be much nicer, I don’t like to mix the different topics on the same page.
I’m going to sit down and create a new design for the site, and rearrange my previous posts. Let’s see how it goes!
I found this yesterday, actually very cool. This would something I’d expect from Flickr, or the Flickr-users, but not from Microsoft. The only problem with this is how to get all the photos, it sure would be cool to hook up this with Flickr
Check out the video here
Vad Tyst Det Blev…
För barnen
Video recordings from some of the talks at Slackathon 2006 (They’re all in Swedish though)
Niklas Hallqvist
Hans Insulander
Måns Nilsson talks about DNSSEC
Some pictures from the event.
Previous post about the Slackathon
This is so cool! I love stopmotion videos.
Via petercooper.co.uk
Finishing updates on some “old” PHP code, pretty nasty experience when you’re used to live happily in the Rails dreamworld
Writing a small application that reads RSS and creates a fullscreen slide show with its content, to “broadcast” information through video on a school.
Rewriting WikImum from scratch, I want it to be more simple and beautiful, code-wise.
Sun X4500 (Disk monster)
4U, 2 dual-core Opterons, 16 GiB RAM, 12-24 TiB internal storage through 48 SATA harddisks. This is just amazing, to get that amout of disks in 4U, and a fully fledged opteron system with lots of RAM.
Sun X4600 (CPU monster)
Also 4U, but instead of disks, this machine is filled with CPUs and RAM. Up to 8 dual-core Opterons and 64 GiB RAM (128 GiB when 4 GiB DIMMs are available).
Sun Blade 8000
Blade server system, 10 systems per chassi, 4 dual-core Opterons with up to 64 GiB RAM for each system.
I’m getting more and more impressed with Sun, their server offerings are just amazing now. First the T1000/T2000 servers, and now this, together with the previous opteron systems. And not to forget, they have one of the most promising operating systems around, Solaris 10 with ZFS.
The X4500
Heard
The Strokes
HIM
In Flames
The Cardigans
Timbuktu & Damn!
Gnarls Barkley
Babyshambles
Phoenix
Within Temptation
The Soundtrack of Our Lives
Martha Wainwright
The Sounds
Ane Brun Duets
Editors
Gogol Bordello
Snook
Bullet For My Valentine
Hello Saferide
Slagsmålsklubben
Deportees
Sirqus Alfon
Missed
I’m from Barcelona
Maia Hirasawa
Datarock
Today I attended the OpenBSD fund-raiser, Slackathon 2006, at Stockholm University. Since I arrived a bit late, 3 hours after it started, I didn’t attend all the talks/presentations. But I did listen to Hans Insulander about OpenBSD development in general and Måns Nilsson on DNSSEC and his work to implement it at KTHNOC, both were very interesting. KTHNOC manages SUNET and NORDUNET, SUNET is the swedish network for universities, lots of bandwidth and usage. Måns Nilsson is a great speaker and his presentation was great. I got a good overview how an DNSSEC-implementation could look like… pretty complex compared to ordinary DNS. :-)
I didn’t socialize that much, since well… I’m a lazy slacker. I left pretty early to work and didn’t attend the dinner. But it was a very nice event, hope something similar will be organized next year. It would be cool with a hackathon combined slackathon, were we users and newbies could attend and “Learn from the masters”. I don’t mean the completly clueless, but like for example: You write a patch for OpenBSD, but since you’re not a regular contributor you don’t really know how to write code that fits in the base. Then you could go through the patch with a “developer”, getting the tips and pointers directly. And hopefully an event like that would produce better code and maybe even some new commiters. =)
(More photos here)
I have to say I’m very pleased with my MacBook, this was my first little adventure with it and it has nothing but impressed me. Battery time is great, the sleep function “just works” and the screen is working really good even with some direct sun light. I did some Rails-development on the way from Norrtälje to Stockholm, and I’m writing this on my way back. Great notebook for work!
When I woke up on Monday, around 11.30 since I hadn’t slept the previous night, a very pleasent surprise waited me down in the hall. Two shiny new MacBooks. :-)
The package was very neat and stylish, just as I tought it would be, since it’s an Apple product.
I’m at Birdie 16 this weekend, check out the pictures!
Apple just announced the 13” MacBook, this is very cool since I’ve been waiting a while for it to come out.
Negatives
The black version costs 1500 SEK more
Intel GMA-chip for the graphics
512 MiB in two memory modules, I need to throw one module away if I’m going to upgrade after delivery
2 GiB RAM is very expensive from factory
All displays are “Glossy”, it’s optional on the MacBook Pro. I’m not sure I’d like a glossy screen
But still, it’s a great machine for it’s 10795 SEK price. I’m probably going to buy a white 1.83 GHz MacBook soon.
Found a very nice BitTorrent client this afternoon, I haven’t been impressed by the “official” Python client so this is very welcome.
btpd is a BitTorrent client consisting of a daemon and client commands,
which can be used to read and/or manipulate the daemon state. The daemon
is capable of running several torrents simultaneously and only uses one
tcp port. It’s fairly low on resource usage and should be perfect for a
torrent distribution site. Efficient downloads and ease of use makes this
client a good choice for the casual user as well.
Be sure to check out btpd!
Yay! There will be a Slackathon at Stockholm University on June 3.
The purpose is the collect lots of donations for the OpenBSD project from the visitors. There will be Swedish OpenBSD celebrities and hopefully lots of OpenBSD users.
Since it’s so close to where I live I have no excuse not to attend. :-)
Niagara vs ftp.heanet.ie Showdown
“ftp.heanet.ie is one of the single busiest webservers in the world. We handle many millions of downloads per day, but unusually for a high-demand site, we do it all from one machine.” […] “Bottom line, the T2000 was able to handle over 3 times the number of transactions per-second and about 60% more concurrent downloads than the current ftp.heanet.ie machine can (a dual Itanium with 32Gb of memory) running identical software.”
The UltraSPARC T1 (Niagara) based Sun Fire T2000 Server (Latest results)
Center for Computing and Communication RWTH Aachen University (I think) have a nice review of the T2000.
Sun’s T2000 “Coolthreads” Server: First Impressions and Experiences
Anandtechs first review of the T2000.
The Sun T1000/T2000 seems to be an awesome server, I’d love to see how Ruby on Rails-applications perform in it.
Don’t forget to read some about the ftp.heanet.ie server too, interesting setup with a heavy loaded Apache installation.
I’ve read a few articles about running Ruby on Rails on servers, how to scale the application by creating a good server architecture. Pretty interesting.
The adventures of scaling, Stage 1 (Q&A) – How to switch a multi milion hits per day website from PHP to Rails, the hardware and initial port.
The adventures of scaling, Stage 2 – How to tune the software and architecture to handle the massive load of a huge community.
Killing me softly: Keeping dispatchers alive – Just noticed this post, haven’t read it yet but it sounds interesting.
Robot Co-op Hardware – What kind of hardware Robot Co-op uses to host 43 things and 43 places
Robot Co-op Software – And what software is used to serve all these requests
Other
Jason Hoffman (Textdrive fame) talks a bit how troublesome it can be to buy servers from Sun and HP.
Rails 1.1 is getting closer with release candidate 1 out. This is great, I’ve been using EdgeRails for about a month in WikImum as I’m moving all Javascript-stuff to RJS. Now I can stay with the “stable” Rails release, but I guess I’ll switch to EdgeRails as soon as some cool new feature comes ut. I’m really glad to see that the goals have not been changed for Rails, this is something DHH has been talking a bit about too, that this release only makes the journey towards that goal a bit more pleseant and fun (RJS).
2006-03-25
16:12
I started Apache with the old config, so it’s working. All services on Phoo should be online now, but the DNS may be lagging a bit behind.
11:20
Dovecot and backups were running at around 03.00 this morning, then I had to sleep some. The initial backup is still running (lots of data). Now I’m going to play around with Apache again, hopefully I can get it to work with FastCGI the way I wan’t, pre-spawned processes for all users that is.
01:44
Mails are flowing from Beaver to Phoo! MySQL-replication is working, Postfix is running. Dovecot will be enabled soon. New backups will be made in a few minutes.
2006-03-24
Apache, Dovecot and MySQL
23:55
Since OpenBSD 3.9 has MySQL 5.0 in ports, and MySQL master and slaves need to run the same MySQL major version, I needed to update Beavers (which runs OpenBSD 3.8) MySQL from 4.1 to 5.0.
It took quite some time to fix this because it:
Took like 30 minutes to build the package
I forgot to run mysql_fix_privilege_tables, ran mysqlcheck on all databases and kept getting “errno 9”, took some time to figure out that the user wasn’t allowed to use more file descriptors.
Had to increase the openfiles-value in /etc/login.conf
Restore a copy of the databases from backup since I hade fiddled around with it
Then start with –skip-grant-tables and run mysql_fix_privilege_tables
But now 5.0 is running och Beaver and Phoo, just have to move around some temporary databases created on Beaver, resync the databases from Phoo and then enable replication.
Postfix and Dovecot should “just work”.
19:20
I’ve got the replication/syncing going now, using rsync and SSH. Now it’s time for MySQL, Postfix and Dovecot!
15:30
Right now I’m looking for a secure way to replicate all the important data on Phoo to Beaver, I’m probably going to use Rsync through SSH or through IPsec, trying to sync the files every 10th minute or so. I just need a small script that checks if there is an active transfer going on, if so it will abort and check again in 10 minutes.
Things that need to replicate to Beaver is
All mails in IMAP storage
Home directories and websites (i.e. /home and /var/www)
Bitlbee accounts
MySQL (Already using MySQL built in replication)
Past tense:
2006-03-23
Package updates and Apache wrestling
I updated all the installed packages and removed some unused ones, because 3.9 isn’t released yet there is no pre-built packages, so I had to build all the packages from ports. It took some time to get everything working because of old libs and such.
Previously we’ve been using Apache with mod_php for PHP5, but I never really liked it, so I thought I’d try and install a better PHP5 environment now that the service is down anyway. I want to use Apache with FastCGI and pre-spawned PHP5-processes. I’m using this, but with Lighttpd, on Hera and I’m pretty happy with the results. Every user has a couple of PHP-processes that can interpret their scripts, no scripts are run as a shared user. Everything is chrooted in /var/www too.
With Apaches mod_fastcgi I couldn’t figure out a way to configure it for pre-spawned FastCGI-processes for all .php files, I could only make it work with specific script (ie. /users/jage/htdocs/index.php). This wasn’t whan I wanted to do, so after whining some on IRC, xevz mentioned mod_perl and using embedded Perl in the httpd.conf, as I’m not a Perl programmer I took a look at mod_ruby instead, but since the documentation was lacking I started reading about mod_perl. I did some tests and couldn’t get it to work then.
I enabled Bitlbee on Phoo and dentarg pointed im.starkast.net back to Phoo. Wonder how many users of our Bitlbee-server will continue to use it for IM, since this massive downtime. I’m going to copy the bitlbee-accounts from Phoo to Beaver every fifth minute and have a copy of the Bitlbee configuration on Beaver so that we can use it as an backup server for all users, in case Phoo wants to cause more trouble.
2006-03-22
The visit
Me and serp visited Phoo in the colocation facility this wednesday. The first thing we did was to connect a monitor the Phoo and see what was going on. What we saw was a normal login-screen without any error messages. I could write my username and press enter, but no password would be asked for, it just “hung”. After we pushed the reset button, we ran some fsck and disabled all the services, we then scoured the logs but didn’t find much of a clue why it had stopped working like that. In the daemon-log it was as if Phoo had been offline for over a month, and in the messages log all I could see was some attempts by ntpd to connect to servers.
We updated OpenBSD from 3.8-stable to 3.9 from CVS, this because we have had some weird reboots with 3.8, hopefully this code will be more stable.
Memtest86 ran a full test on all the RAM (2GiB) and no errors or faults were found.
We spent several hours trying to flash the IPMI card without success, we decided to bring the card home with us, hopefully we can get it replaced.
This trip took us a whole day, when I came home all I did was reboot the server with a freshly built kernel and then went to bed.
BSDTalk has been producing a steady stream of BSD podcasts for a few months now. It’s a great resource for easily digested information about BSD.
Yesterday I listened to the interview with Henning Brauer and Matthew Dillon, They were both pretty interesting, even though I’ve heard most of the topics before.
Video
Theo de Raadt has been interviewed by Howard Green on the The Business Show. Here’s the interview without commercials and here’s the complete show. (From Undeadly)
I also found a video of Matthew Dillons presentation at BayLISA 2005 on Google Video, slides here. (From DragonFly BSD Digest)
I was forced by dentarg to take some photos of my desktop today, and while I had the camera up I snapped a few shots of my other machines as well.
Here’s some more photos
Carson Workshops has published the talks from the Future of Web Apps conference as podcasts.
Tom Coates´ and Joshua Schachters talks was very interesting. Haven’t had time to listen to all of them, Ryan Carson and David Heinemeier Hanssons will probably be very interesting as well.
ToS
When SSH is used for an interactive login session, the ToS-field in the IP-headers will be set to “minimize delay” (0×10), PF can use this information to place the packets in a different queue. SFTP and SCP will set the ToS to “maximize throughput” (0×8) and when SSH are being used as a transport for other protocols, such as CVS or Rsync, no ToS-value will be set at all.
tcpdump1 output from an interactive login session.
0.0.0.0.27859 > 0.0.0.0.22: tcp 0 (DF) [tos 0x10] (ttl 64, id 8913, len 52)
Output from an SFTP file transfer, notice the different ToS-value.
0.0.0.0.8321 > 0.0.0.0.22: tcp 1448 (DF) [tos 0x8] (ttl 64, id 19794, len 1500)
PF
When using the queue keyword in PF, one or two queues can be specified. If the second queue is specified, it will be used for packets with the ToS-header set to “minimize delay” or if it is being used in conjunction with “keep state”, the ACK packets will be assign to that queue.
In my ruleset, I use this rule to assign the interactive SSH to another queue, with a higher priority.
pass out on $ext_if $tcp to port ssh flags S/SA queue (std_out, ssh_out)
With this I can use SCP and SFTP without having to worry about chocking other traffic.
One thing I have noticed is that the first few packets being sent when starting an SSH-connection will not have the ToS-field set. So the SSH-initiation could be a bit “slow” when you’re upstream bandwidth is full. I don’t know how to solve this yet, though I don’t think it cause any major pain.
Another thing I’ve just noticed is that when I’m downloading through SCP/SFTP, the ACK packets are put in the wrong queue, they’re in the ssh_out-queue but I want them in the ack_out-queue. Have to ponder that one a bit more.
Here’s my complete ruleset.
Footnote
1 The -tvqn parameters were given to tcpdump(8)
Postfix superintendent, postsuper, is the command to use for maintenance jobs on the queue.
With the option -r
Since Phoo is still having problems, I needed to use this to redirect the mails that were already in the deferred queue at the backup MX, mail that had arrived before I had added an override redirect.
Read more in the Postfix manual
Phoo.starkast.net stopped responding to requests the night between Sunday and Monday. Don’t really know what happened, when I woke up my SSH-sessions were dead and I couldn’t reconnect. HTTP and IMAP was dead too. I didn’t have time to look into it at the time, as I had school on the morning and work later.
On Monday dentarg mailed the administrator of the firewall that sits in front of the server, explaining the situation. He checked the firewall logs and said that Phoo was flooding some packets and the firewall thought it was an internal DoS and blocked it. He then added some firewall rules that allowed my IP to connect without matching any firewall rules, but I couldn’t initiate a SSH-session. When I looked at the traffic, Phoo only managed to respond with a SYN+ACK to my SYN packet, and then kept quiet. The TCP/IP-stack on Phoo seems to be working, but the sshd can’t respond. The weird thing is that my IRC-client was still connected to the IRC-networks on the morning, and Luddes IRC-client was connected for 48h and then got a “Ping timeout”. I’ve also noticed that the DNS-server is still responding.
This is a bit annoying, since I do my developing on Phoo, and some of us has our mail there. Regarding mail, our other server Beaver.starkast.net is a backup MX for all the domains, and I’ve changed Postfix so that all the aliases that forwards to external addresses can do so right away from Beaver. My two main e-mail accounts are forwarding all new mail to my Gmail-account now.
One thing I haven’t figured out with Postfix, is how to re-run all the mails that’s in the relay queue. New mails will be forwarded to my Gmail-account, but the mail recieved before I added the virtual aliases still tries to be delivered to Phoo since the mail has already been added to the relay queue. It would be great if I could pull out the mails and send it to the pickup process (IIRC).
We’re probably forced to visit the co-location facility and check what’s wrong. We could pay 800 SEK for someone at the facility to press the reset button on Phoo, but since we don’t know what’s wrong, it just feels like a waste of money.
Wonder if using watchdog would have helped.
Update
We have decided not to pay 800 SEK and reboot Phoo, we will try and visit Phoo in the co-location facility. When the visit will take place we do not know, probably in a few weeks from now.
I just stumbled upon a cool video: I use magic: morning – See it.
I’ve finally switched to production mode on the blog, it has been lying around idle on an obscure port for almost a month now.
It’s not really “optimal” yet though, since I haven’t figured out a good way to handle FastCGI + suexec on Phoo I moved the blog to my Sun Blade 100 machine in the basement, and pointed blog.jage.se to it. But I don’t like the sub-domain, I want everything to be accessible on jage.se but this is better than nothing.
I have also switched back to “production mode” from the christmas “slack mode”.
I spent the christmas holiday at my grand parents, it was really nice.
Second week in January I went to Åre with some friends, one week of constant Snowboarding without Internet-access was great! Hope I will be able to go on at least one more snowboard trip this winter.
Interview with Jason Fried, the man behind 37signals.
The Ruby on Rails podcast also had some good shows lately, I really enjoyed the ones with:
Dan Benjamin
Chad Fowler
ODEO Rabble
Today I started using Switchtower, the Ruby on Rails application deployment software, and I’m truly impressed with the powerfulness of it. I’ve never really used deployment systems before, I’ve always done it manually, but now I’m hooked.
Now I can keep my focus on the code and just execute rake deploy when I think a feature is ready.
Have to check out “Gauge”, the application monitor system for rails, some day too. Haven’t heard much about it since before the 1.0 launch. It would be great to have a real-time monitor system for all the applications. Mostly because I can be notified if there seems to be a problem, but also to generate cool graphs! :-)
After a couple of failed attempts, I have created a blog layout that I can live with!
I just created a minimal theme for Typo, rewrote the Article#Archives method, added a new route and wrote some helpers. Hope it won’t be too much of a hassle to patch future typo releases with my hacks tough. When I last attempted to use Typo for a blog, I got really scared of all the functionallity, that it would be too “bloated”, but I really don’t have time to write a blog software from scratch, so I’ll just have to mentally ignore all the code that I’m not utilizing. :-)
I’m really proud of the site layout, it has “just enough” stuff and nice typography.