File systems
ZFS
zetabyte file system It is not part of linux distributions but you can use the user space version zfs-fuse which seems to work fine.
iscsi with zfs
You can use zfs fuse on iscsi disks, however i had problems with the array after rebooting. Mostly if you see that all or most disk are reporting corrupt data or missing labels what probably is the case is you created the pool like :
| create pool | |
|---|---|
However iscsi does not guarantee that sdb will be sdb after a reboot. As a matter of fact you can just as well have that problem with you BIOS !! The solution is to use the alternate paths for disks. For instance your disks are listed in /dev by id, path or uuid.
disk id
| disk id | |
|---|---|
disk path
| disk path | |
|---|---|
disk uuid
this is a real unique disk id
| disk uuid | |
|---|---|
disk label
Sometimes also by label
I used by path for the iscsi disks, they looked like this
So that's specific enough not to get switched around.
iscsi
Iscsi is scsi over a network. For linux admin's this will be rather explanatory : '''you can fdisk a disk on another machine''' And of course you can do almost anything ales you would with anything you can 'fdisk'. I myself used it to create a zfs (-fuse) raidz array over multiple machines ;-). So it might be best to describe how to do that ?
setting up iscsi
Of course you need an operating system to run the software, so i chose a simple debian lenny netinst. From there i will be very terse and am for the commands issued, but one thing in advance for installation iscsitarget is the server side software, so to provide iscsi disks. open-iscsi is the client side, so it' to use iscsi disks. In iscsi the client is often called initiator, and the server the target.
server
The server is called iSCSI Enterprise Target Daemon, and called ietd. The admin command therefore also is called ietadm.
| server | |
|---|---|
The result from that last command will give you a clue about what modules to install, for instance mine was :
| iscsitarget | |
|---|---|
It will warn about not being started up, because it is disabled in /etc/default/iscsitarget. First solve that by editing the file or if ye'r really lazy :
| enable | |
|---|---|
At your own risk, it is currently the only line there so it works fine but you are warned. But you have to alter /etc/ietd.conf it has some reasonable defaults but of course your disks should be defined. Take the example and alter/copy it. I did it without authentication, so my only alterations to the default section where :
| /etc/ietd.conf | |
|---|---|
The iqn is the iscsi qualified name, it has to be globally and chronologically unique. So it was made to look like this :
| iqn | |
|---|---|
The yyyy.mm date is any date that you owned the domain you use. The idea is that you have an unique identifier for an Internet domain at a certain time and it could be sold to another one in which case the date kicks in. I just did the current date because i know we still own almende.net. Play around with the option if you like, i did not. Of course you decide which partitions (or even file it seems ?) you want to export as a target. I made two equally sized partitions because they are going to act as part a zfs array later on. Now restart the target and watch for any errors
| restart | |
|---|---|
On the server you can issue some commands like:
To view the status, or go on to the client section to view over the network.
client
The client is also called the initiator, and since you tend to be either a client or a server, there is a different package to install :
| client | |
|---|---|
Now to spare you some work alter the startup in /etc/iscsi/iscsi.conf to 'automatic' see below. And run this command to find iscsi targets :
| iscsiadm | |
|---|---|
- -m means mode so discovery mode
- -t is type, sendtarget or st for short
- -p is portal , so give the ip address of a target
It will return :
| output | |
|---|---|
After this, you are logged in and discovery will return :
After doing all 3 machine it looks like :
| output | |
|---|---|
With 'node' mode you can view all nodes :
Also the settings for each node are made permanent in /etc/iscsi/nodes, there is a directory there for each of the nodes. But still we need to login in to the remote target, so use :
| remote | |
|---|---|
If you use no authentication, like i did above, you can now do :
| show | |
|---|---|
And hey you get an extra harddisk /dev/sdb !! You can do this for all targets but to automate it at boot time see next section.
auto startup
Alter /etc/iscsi/iscsi.conf :
| /etc/iscsi/iscsi.conf | |
|---|---|
Change the default from manual to automatic, this only affects targets discovered '''after''' you set this, so here is a manual way of doing it. You can edit the node files yourself, but this is the correct way of doing it :
| automatic | |
|---|---|
You may notice there is also a 'node.startup=manual' in the node files , but that does not seem to affect startup. And nowhere in the entire documentation and Internet is there anyone mentioning the difference between the two so i gave up. See [[Zfs_linux]] for how to create a useful system out of these. But here is the command i used for 3 machine with each 2 disks :
| zfs in iscsi | |
|---|---|