RPC Internals

From GlusterFS Documentation
Jump to navigationJump to search
Home


Contents[edit | edit source]

Overview[edit | edit source]

Gluster uses a transport independent RPC mechanism for communications between cluster processes. Within Gluster the two primary consumers of RPC services are:

  • glusterd : used for communications between glusterd management processes.
  • protocol/client and protocol/server translators. This translator pair is inserted in a translator stack to enable a a translator stack to span multiple nodes, such as when a brick is served remotely. The protocol/client and protocol/server translators implement the full translator API, including all allowable FOPS, enabling it to be transparently inserved within a volspec.

RPC Usage[edit | edit source]

Below RPC usage table is based on references to rpc_clnt_new() ` and `rpcsvc_init()

Module Client Server
api Yes No
cli Yes No
glusterfsd Yes Yes
xlators/mgmt/glusterd Yes Yes
xlators/nfs/server Yes Yes
xlators/protocol/Client Yes No
xlators/protocol/server No Yes

protocol/client Translator[edit | edit source]

Client Translator volume specification[edit | edit source]

volume my_vol-client-0
    type protocol/client
    option remote-host 192.168.122.125
    option remote-subvolume /glusterdata/brick125
    option transport-type tcp
end-volume

protocol/client Internals[edit | edit source]

Source may be found at: xtalors/protocol/client

protocol/server Translator[edit | edit source]

Server Translator volume specification[edit | edit source]

volume my_vol-server
    type protocol/server
    option transport-type tcp
    option auth.login./glusterdata/brick125.allow 79431724-6ea3-482f-8d2e-7654d8934d83
    option auth.login.79431724-6ea3-482f-8d2e-7654d8934d83.password be6b8ca2-89fb-4223-b521-d976e3c5f6cd
    option auth.addr./glusterdata/brick125.allow *
    subvolumes /glusterdata/brick125
end-volume

protocol/server Internals[edit | edit source]

Source may be found at: xlators/protocol/server

Auth[edit | edit source]

Authenticated Volspecs[edit | edit source]

Example of client-side authenticated connection:

volume my_vol-client-0
    type protocol/client
    option remote-host 192.168.122.125
    option remote-subvolume /glusterdata/brick125
    option transport-type tcp
    option username 79431724-6ea3-482f-8d2e-7654d8934d83
    option password be6b8ca2-89fb-4223-b521-d976e3c5f6cd
end-volume

Example of server-side authenticated connection:

volume my_vol-server
    type protocol/server
    option transport-type tcp
    option auth.login./glusterdata/brick125.allow 79431724-6ea3-482f-8d2e-7654d8934d83
    option auth.login.79431724-6ea3-482f-8d2e-7654d8934d83.password be6b8ca2-89fb-4223-b521-d976e3c5f6cd
    option auth.addr./glusterdata/brick125.allow *
    subvolumes /glusterdata/brick125
end-volume

Auth Internals[edit | edit source]

Source may be found at: `xlators/protocol/auth<code> `

glusterd use of RPC[edit | edit source]

RPC calls within glusterd are localized to the <code>gllusterd-rpc-p[s.c file, which includes references to shared RPC via the collowing includes:

  • RPC: rpc-clnt.h and protocol-common.h
  • Wire encoding: glusterd1-xhr.h, cli1-xdr.h and xdr-generic.h.

RPC related volume options for glusterd:

struct volume_options options[] = {
...
        { .key   = {"transport-type"},
          .type  = GF_OPTION_TYPE_ANY,
        },
        { .key   = {"transport.*"},
          .type  = GF_OPTION_TYPE_ANY,
        },
        { .key   = {"rpc-auth.*"},
          .type  = GF_OPTION_TYPE_ANY,
        },
        { .key   = {"rpc-auth-allow-insecure"},
          .type  = GF_OPTION_TYPE_BOOL,
        },
...
};

RPC initialization snippet in glusterd.c init() code:

        ret = rpcsvc_register_notify (rpc, glusterd_rpcsvc_notify, this);
...
        ret = rpcsvc_create_listeners (rpc, this->options, this->name);
...
        ret = glusterd_program_register (this, rpc, &gd_svc_peer_prog);
...
        ret = glusterd_program_register (this, rpc, &gd_svc_cli_prog);
...
        ret = glusterd_program_register (this, rpc, &gd_svc_mgmt_prog);
...
        ret = glusterd_program_register (this, rpc, &gluster_pmap_prog);
...
        ret = glusterd_program_register (this, rpc, &gluster_handshake_prog);
...

Background on functions called:

  • rpcsvc_register_notify (rpc, glusterd_rpcsvc_notify, this): call to core RPC notify registration routine located in rpc/rpc-lib/src/rpcsvc.c
  • rpcsvc_create_listeners (rpc, this->options, this->name): call to core RPC listener registration routine located in rpc/rpc-lib/src/rpcsvc.c
  • glusterd_program_register (this, rpc, &XXX_prog): calls core RPC program registration routine (rpcsvc_program_register) located in rpc/rpc-lib/src/rpcsvc.c

Specific programs registered:

  • gd_svc_peer_prog
  • gd_svc_cli_prog
  • gd_svc_mgmt_prog
  • gluster_pmap_prog
  • gluster_handshake_prog

sample glusterd volspec[edit | edit source]

Sample volspec file for glusterd including reference to rpc. Can be found at `/etc/glusterfs/glusterd.vol `

volume management
     type mgmt/glusterd
     option working-directory /var/lib/glusterd
     option transport-type socket,rdma
     option transport.socket.keepalive-time 10
     option transport.socket.keepalive-interval 2
     option transport.socket.read-fail-log off
end-volume

Other functions called:

  • rpc_clnt_transport_unix_options_build (&options, socketpath);
  • glusterd_submit_request (peerinfo→rpc, &req, frame, peerinfo→peer, GLUSTERD_PROBE_QUERY, NULL, this, glusterd_probe_cbk, (xdrproc_t)xdr_gd1_mgmt_probe_req);
    • rpc_clnt_submit (rpc, prog, procnum, cbkfn, &iov, count, NULL, 0, iobref, frame, NULL, 0, NULL, 0, NULL);
  • glusterd_submit_reply (req, cli_rsp, NULL, 0, NULL, xdrproc);
    • glusterd_serialize_reply (req, arg, &rsp, xdrproc);
    • rpcsvc_submit_generic (req, &rsp, 1, payload, payloadcount, iobref);
  • xdr_to_generic (*iov, &rsp, (xdrproc_t)xdr_gd1_mgmt_probe_rsp);

Libraries[edit | edit source]

rpc-lib[edit | edit source]

rpc-clnt.h[edit | edit source]

struct rpc_clnt *rpc_clnt_new (dict_t *options, glusterfs_ctx_t *ctx,
                               char *name, uint32_t reqpool_size);

int rpc_clnt_start (struct rpc_clnt *rpc);

int rpc_clnt_register_notify (struct rpc_clnt *rpc, rpc_clnt_notify_t fn,
                              void *mydata);

int rpc_clnt_submit (struct rpc_clnt *rpc, rpc_clnt_prog_t *prog,
                     int procnum, fop_cbk_fn_t cbkfn,
                     struct iovec *proghdr, int proghdrcount,
                     struct iovec *progpayload, int progpayloadcount,
                     struct iobref *iobref, void *frame, struct iovec *rsphdr,
                     int rsphdr_count, struct iovec *rsp_payload,
                     int rsp_payload_count, struct iobref *rsp_iobref);

struct rpc_clnt *
rpc_clnt_ref (struct rpc_clnt *rpc);

struct rpc_clnt *
rpc_clnt_unref (struct rpc_clnt *rpc);

int rpc_clnt_connection_cleanup (rpc_clnt_connection_t *conn);

void rpc_clnt_set_connected (rpc_clnt_connection_t *conn);

void rpc_clnt_unset_connected (rpc_clnt_connection_t *conn);
void rpc_clnt_reconnect (void *trans_ptr);

void rpc_clnt_reconfig (struct rpc_clnt *rpc, struct rpc_clnt_config *config);

int rpcclnt_cbk_program_register (struct rpc_clnt *svc,
                                  rpcclnt_cb_program_t *program, void *mydata);

int
rpc_clnt_transport_unix_options_build (dict_t **options, char *filepath);

void
rpc_clnt_disable (struct rpc_clnt *rpc);

rpc-svc.h[edit | edit source]

extern int
rpcsvc_program_register (rpcsvc_t *svc, rpcsvc_program_t *program);

extern int
rpcsvc_program_unregister (rpcsvc_t *svc, rpcsvc_program_t *program);

extern int32_t
rpcsvc_create_listeners (rpcsvc_t *svc, dict_t *options, char *name);

void
rpcsvc_listener_destroy (rpcsvc_listener_t *listener);

extern int
rpcsvc_program_register_portmap (rpcsvc_program_t *newprog, uint32_t port);

extern int
rpcsvc_register_portmap_enabled (rpcsvc_t *svc);

extern rpcsvc_t *
rpcsvc_init (xlator_t *xl, glusterfs_ctx_t *ctx, dict_t *options,
             uint32_t poolcount);

int
rpcsvc_register_notify (rpcsvc_t *svc, rpcsvc_notify_t notify, void *mydata);

int
rpcsvc_unregister_notify (rpcsvc_t *svc, rpcsvc_notify_t notify, void *mydata);

int
rpcsvc_submit_message (rpcsvc_request_t *req, struct iovec *proghdr,
                       int hdrcount, struct iovec *payload, int payloadcount,
                       struct iobref *iobref);

int
rpcsvc_submit_generic (rpcsvc_request_t *req, struct iovec *proghdr,
                       int hdrcount, struct iovec *payload, int payloadcount,
                       struct iobref *iobref);

extern int
rpcsvc_error_reply (rpcsvc_request_t *req);

extern int
rpcsvc_transport_peername (rpc_transport_t *trans, char *hostname, int hostlen);

extern inline int
rpcsvc_transport_peeraddr (rpc_transport_t *trans, char *addrstr, int addrlen,
                           struct sockaddr_storage *returnsa, socklen_t sasize);

extern int
rpcsvc_transport_peer_check (dict_t *options, char *volname,
                             rpc_transport_t *trans);

extern int
rpcsvc_transport_privport_check (rpcsvc_t *svc, char *volname,
                                 rpc_transport_t *trans);

extern int rpcsvc_submit_vectors (rpcsvc_request_t *req);

extern int rpcsvc_request_attach_vector (rpcsvc_request_t *req,
                                         struct iovec msgvec, struct iobuf *iob,
                                         struct iobref *ioref, int finalvector);

extern int
rpcsvc_auth_request_init (rpcsvc_request_t *req);

extern int
rpcsvc_auth_init (rpcsvc_t *svc, dict_t *options);

extern int
rpcsvc_auth_transport_init (rpc_transport_t *xprt);

extern int
rpcsvc_authenticate (rpcsvc_request_t *req);

extern int
rpcsvc_auth_array (rpcsvc_t *svc, char *volname, int *autharr, int arrlen);

extern gid_t *
rpcsvc_auth_unix_auxgids (rpcsvc_request_t *req, int *arrlen);

extern int
rpcsvc_combine_gen_spec_volume_checks (int gen, int spec);

extern char *
rpcsvc_volume_allowed (dict_t *options, char *volname);

int rpcsvc_callback_submit (rpcsvc_t *rpc, rpc_transport_t *trans,
                            rpcsvc_cbk_program_t *prog, int procnum,
                            struct iovec *proghdr, int proghdrcount);

int
rpcsvc_transport_unix_options_build (dict_t **options, char *filepath);

int
rpcsvc_set_allow_insecure (rpcsvc_t *svc, dict_t *options);

int
rpcsvc_auth_array (rpcsvc_t *svc, char *volname, int *autharr, int arrlen);

char *
rpcsvc_volume_allowed (dict_t *options, char *volname);

rpcsvc_vector_sizer
rpcsvc_get_program_vector_sizer (rpcsvc_t *svc, uint32_t prognum,
                                 uint32_t progver, uint32_t procnum);

rpc-transport[edit | edit source]

socket[edit | edit source]

rdma[edit | edit source]

xdr[edit | edit source]


Home