Gluster Internals- Jeff Darcy

From GlusterFS Documentation
Jump to navigationJump to search
Home

See also:

Contents[edit | edit source]

Getting Started: Jeff Darcy Tutorials[edit | edit source]

An excellent set of tutorials can be found on hekafs.org blog

Translator API[edit | edit source]

External APIs[edit | edit source]

GlusterFS uses Extended Attributes as a mechanism for external interaction with translators. Unlike the static values of a transitional Posix file or directory access, Glusterfs xattr values can be dynamically generated and setxattr can be thought of as a form of method invocation. The HekaFs Blog contains a good discussion of GlusterFS Extended Attributes.

Glusterfs extended attributes include:

  • trusted.glusterfs.afr
    • trusted.glusterfs.afr.data-pending
    • trusted.glusterfs.afr.metadata-pending
    • trusted.glusterfs.afr.entry-pending
  • client
    • trusted.glusterfs.client-connect
    • trusted.glusterfs.client-disconnect
    • trusted.glusterfs.clientlk-dump
  • trusted.glusterfs.createtime
  • trusted.glusterfs.dht
    • trusted.glusterfs.dht.linkto
  • trusted.glusterfs.node-uuid
  • trusted.glusterfs.pathinfo
  • trusted.glusterfs.pump
    • trusted.glusterfs.pump-path
    • trusted.glusterfs.pump-source-complete
    • trusted.glusterfs.pump-sink.complete
  • trusted.glusterfs.quota
    • trusted.glusterfs.quota.dirty
    • trusted.glusterfs.quotq.size
    • trusted.glusterfs.quota.*.contri
    • trusted.glusterfs-quota-du
  • trusted.glusterfs.set-log-level
    • trusted.glusterfs.*.set-log-level
      • trusted.glusterfs.fuse.set-log-level
  • trusted.glusterfs.syslog
  • trusted.glusterfs.test
  • trusted.glusterfs.version
  • volume
    • trusted.glusterfs.volume-id
    • trusted.glusterfs.volume-mark
  • trusted.glusterfs.xtime or system.glusterfs.xtime
    • trusted.glusterfs.*.xtime

Other extended attributes include:

  • trusted.afr.createtime
  • trusted.afr.version
  • trusted.afr.volume
  • trusted.distribute.linkinfo
  • trusted.gfid
  • trusted.limit.list
  • trusted.quota*
  • trusted.testing
  • trusted.xattr-test

Internal APIs[edit | edit source]

Gluster translator APIs are contained in libglusterfs. Useful documentation of the various APIs can be found on Hekafs.org.

APIs and support functions includes:

  • Dispatch Tables and Default Functions
  • STACK_WIND and STACK_UNWIND
  • Per Request Context
  • Inode and File Descriptor Context
  • Dictionaries and Translator Options
  • logging
  • Child Enumeration and Fan Out

Gluster file and directory operations closely model those of FUSE. The FUSE sourceforge site contains useful information on definition of many of these operations.

Gluster CLI[edit | edit source]

Hooks Mechanism[edit | edit source]

Hooks Mechanisms:

Hooks — from: glusterfs / xlators / mgmt / glusterd / src / glusterd-hooks.c

#define EMPTY ""
char glusterd_hook_dirnames[GD_OP_MAX][256] =
{
        [GD_OP_NONE] = EMPTY,
        [GD_OP_CREATE_VOLUME] = "create",
        [GD_OP_START_BRICK] = EMPTY,
        [GD_OP_STOP_BRICK] = EMPTY,
        [GD_OP_DELETE_VOLUME] = "delete",
        [GD_OP_START_VOLUME] = "start",
        [GD_OP_STOP_VOLUME] = "stop",
        [GD_OP_DEFRAG_VOLUME] = EMPTY,
        [GD_OP_ADD_BRICK] = "add-brick",
        [GD_OP_REMOVE_BRICK] = "remove-brick",
        [GD_OP_REPLACE_BRICK] = EMPTY,
        [GD_OP_SET_VOLUME] = "set",
        [GD_OP_RESET_VOLUME] = EMPTY,
        [GD_OP_SYNC_VOLUME] = EMPTY,
        [GD_OP_LOG_ROTATE] = EMPTY,
        [GD_OP_GSYNC_SET] = EMPTY,
        [GD_OP_PROFILE_VOLUME] = EMPTY,
        [GD_OP_QUOTA] = EMPTY,
        [GD_OP_STATUS_VOLUME] = EMPTY,
        [GD_OP_REBALANCE] = EMPTY,
        [GD_OP_HEAL_VOLUME] = EMPTY,
        [GD_OP_STATEDUMP_VOLUME] = EMPTY,
        [GD_OP_LIST_VOLUME] = EMPTY,
        [GD_OP_CLEARLOCKS_VOLUME] = EMPTY,
        [GD_OP_DEFRAG_BRICK_VOLUME] = EMPTY,
};

Configuration and Vol Files[edit | edit source]

Sample Glusterd Vol File[edit | edit source]

Used to configure the glusterd management agent. Can be found at `/etc/glusterfs/glusterd.vol `

volume management
     type mgmt/glusterd
     option working-directory /var/lib/glusterd
     option transport-type socket,rdma
     option transport.socket.keepalive-time 10
     option transport.socket.keepalive-interval 2
     option transport.socket.read-fail-log off
end-volume

Sample Client-side Vol File[edit | edit source]

Used to configure each volume available to a client. Files can be found in /var/lib/glusterd/vols/vol_name/ subdirectory

volume my_vol-client-0
    type protocol/client
    option remote-host 192.168.122.125
    option remote-subvolume /glusterdata/brick125
    option transport-type tcp
end-volume

volume my_vol-client-1
    type protocol/client
    option remote-host 192.168.122.236
    option remote-subvolume /glusterdata/brick236
    option transport-type tcp
end-volume

volume my_vol-dht
    type cluster/distribute
    subvolumes my_vol-client-0 my_vol-client-1
end-volume

volume my_vol-write-behind
    type performance/write-behind
    subvolumes my_vol-dht
end-volume

volume my_vol-read-ahead
    type performance/read-ahead
    subvolumes my_vol-write-behind
end-volume

volume my_vol-io-cache
    type performance/io-cache
    subvolumes my_vol-read-ahead
end-volume

volume my_vol-quick-read
    type performance/quick-read
    subvolumes my_vol-io-cache
end-volume

volume my_vol-md-cache
    type performance/md-cache
    subvolumes my_vol-quick-read
end-volume

volume my_vol
    type debug/io-stats
    option latency-measurement off
    option count-fop-hits off
    subvolumes my_vol-md-cache
end-volume

There is often a trusted version of the vol file located in the same directory. Below is an example of the additional information in protocol/client entry for the trusted variant.

volume my_vol-client-0
    type protocol/client
    option remote-host 192.168.122.125
    option remote-subvolume /glusterdata/brick125
    option transport-type tcp
    option username 79431724-6ea3-482f-8d2e-7654d8934d83
    option password be6b8ca2-89fb-4223-b521-d976e3c5f6cd
end-volume

Sample Server-side Vol File[edit | edit source]

Used to configure each brick in the trusted storage pool. Files can be found in ` /var/lib/glusterd/vols/vol_name/` subdirectory

volume my_vol-posix
    type storage/posix
    option directory /glusterdata/brick125
    option volume-id 5e2d14d7-8352-46ca-8b7f-b965c277871f
end-volume

volume my_vol-access-control
    type features/access-control
    subvolumes my_vol-posix
end-volume

volume my_vol-locks
    type features/locks
    subvolumes my_vol-access-control
end-volume

volume my_vol-io-threads
    type performance/io-threads
    subvolumes my_vol-locks
end-volume

volume my_vol-index
    type features/index
    option index-base /glusterdata/brick125/.glusterfs/indices
    subvolumes my_vol-io-threads
end-volume

volume my_vol-marker
    type features/marker
    option volume-uuid 5e2d14d7-8352-46ca-8b7f-b965c277871f
    option timestamp-file /var/lib/glusterd/vols/my_vol/marker.tstamp
    option xtime off
    option quota off
    subvolumes my_vol-index
end-volume

volume /glusterdata/brick125
    type debug/io-stats
    option latency-measurement off
    option count-fop-hits off
subvolumes my_vol-marker
end-volume

volume my_vol-server
    type protocol/server
    option transport-type tcp
    option auth.login./glusterdata/brick125.allow 79431724-6ea3-482f-8d2e-7654d8934d83
    option auth.login.79431724-6ea3-482f-8d2e-7654d8934d83.password be6b8ca2-89fb-4223-b521-d976e3c5f6cd
    option auth.addr./glusterdata/brick125.allow *
    subvolumes /glusterdata/brick125
end-volume

Brick file[edit | edit source]

Additional information on each brick can be found in /var/lib/glusterd/vols/vol_name/bricks subdirectory

hostname=192.168.122.125
path=/glusterdata/brick125
listen-port=24010
rdma.listen-port=0
decommissioned=0


Home