FhGFS is a awesome distributed parallel filesystem. Its simple and so powerful with a RDMA backend that knocks the performance socks off other distributed filesystems, IMHO. Today however I needed to migrate the metadata server from one host ( a machine on loan from Dell ) to another server. FhGFS provides a tool on the Management node called " fhgfs-ctl " which manages your FHGFS environment. The migration options which are listed however only pertain to the storage node types and not the metadata node types. FhGFS also do not provide usable, warm fuzzy feeling documentation for the migration of the metadata server. So I plan to document my migration plan step by step for my own sanity and others. Fraunhofer are welcome to critize my guide. <disclaimer> The step by step guide listed here is my own work and not that from Fraunhofer. I am also not affiliated with Fraunhofer and suggest that all work is performed in a sandpit environment first.</disclaimer>
NB: You will require a maintenance window to perform the migration.
1. Stop all clients / storage nodes / metadata services. Do not shutdown the Management service as we need to make use of the "fhgfs-ctl " tool to manage the environment.
2. Backup your metadata environment - http://www.fhgfs.com/wiki/FAQ#ea_backup
- /beegfs_meta ( This contains your inodes/dentries/ )
- /etc/fhgfs/fhgfs-meta.conf
- Make sure that you are backing up the Extended Attributes. Please read the URL above.
2. Install the FhGFS Metadata RPM and FhGFS Client RPM on the Metadata server.
3. Restore the metadata (inodes/dentries/) and configuration files from the previous server. Do not forget to change the hostname in the /etc/fhgfs/fhgfs-meta.conf file to the new server.
4. Once you have successfully restored the file structure its time to decommission the current metadata server. In order todo this we need to determine what the NodeID and TargetID is.
Execute the following to obtain the NodeID - " fhgfs-ctl --listnodes --nodetype=meta "
Execute the following to obtain the TargetID - " fhgfs-ctl --listtargets "
Match the NodeID to the TargetID
5. Remove the node now from the FhGFS cluster via the management tool
Execute the following to remove the NodeID - " fhgfs-ctl --removenode --nodetype=meta <NodeID> "
Execute the following to remove the TargetID - " fhgfs-ctl --unmaptarget <TargetID> "
NB: Set the following options in the fhgfs-mgmt.conf. This ensure that new servers are allowed to register. You are welcome to return it to your preferred setting once your FhGFS clients are mounted.
storeAllowFirstRunInit = true
sysAllowNewServers = true
6. Since we already populated /etc/fhgfs/fhgfs-meta.conf with the new hostname of the server, restarting the management service will commission the new metadata service. Confirm that the Management service is running by checking its status. On confirmation, start the Metadata server. Executing " fhgfs-ctl --listnodes --nodetype=meta " should list the new MD server.
7. Start up all storage servers and ensure that they are all running.
8. Start the helperd and client services.
Thanks to Dell (Marc / Lyle ) for negotiating the loan of the equipment for our FhGFS POC. We are almost ready to expand and rubber stamp as a production environment.