A panic with no /dev/null on your Linux server

Very early in my professional system administration career, I was lucky (unlucky?) enough to take over a Mandrake server running an alpha release of Samba 2.2.x that was running as a NT domain Primary Domain Controller (PDC).  The system actually ran pretty well, with other Windows NT4 servers attached to the domain and authenticating users to those servers from about 90-100 Windows 95 & 98 workstations throughout the building.

Because the system was running well and working for what we needed, in a production setting, I didn’t tend to tinker with the system, because… it was a production environment.  Over time, my employer started to get Windows 2000 machines and I was testing out how to image and deploy Windows 2000 computers.  In the process, I was creating new machine names over and over and attaching those to the domain.

One day, as I looking at the lists of users on the Mandrake server, I was noticing all the linux users created through attaching those test Windows 2000 workstations to the domain, which is perfectly normal operation for a Samba NT4 style PDC.   Looking at the list of users, I decided to delete all the linux users matching with Window 2000 computer accounts that I knew were gone.

So, I went through and deleted about 10-15 linux users after double and triple checking to verify those accounts weren’t in use and then I deleted those using the command below, which will delete the user account and remove the user’s home directory.

userdel -rf workstation01
userdel -rf workstation02

And that’s where things started going south…. Users weren’t able to log in, printing quit, everything just quit working.  Fortunately, the day was finished, so I was able to tell the couple of people working, I was looking at it and I’d have it working by the next morning. I didn’t add what was running through my head which was “hopefully”

So, after about 6 hours of diagnostic work of looking at logs, reading error messages, and keeping the panic down to a minimum, I realized as the Samba service was creating workstation accounts, the Samba  configuration was setting the linux workstation account with a home directory of /dev/null  Webmin wouldn’t delete /dev/null if it was set as a user’s home directory… would it?

Yes… yes, it would.

So, I had the problem, now to find the solution.  After a hour of trying to figure out how to recreate /dev/null  and not having any success, I called a friend with more experience to come and take a look.  It took him 1 minute to fix the problem and 3 minutes to show me the man page describing how to do it.

rm /dev/null
mknod -m 0666 /dev/null c 1 3

We deleted the incorrect normal file /dev/null and used mknod to create the normal, special device file of /dev/null and the moment we did that, the test printer at the end of the work bench started printing out all the queued test prints I had sent it.

I thanked my friend, who had just earned many lunches on me, and spent 15 minutes calming down and called it a night.   The Mandrake’s kernel never panicked, but I certainly had for an entire evening.