scientificlinuxforum.org QR code
Scientific Linux Forum.org



  Reply to this topicStart new topicStart Poll

> SecureNFS - Trouble migrating from SL5.5 to SL6, rpc.gssd errors
sworkhard
 Posted: Jun 16 2011, 08:33 PM
Quote Post


SLF Newbie


Group: Members
Posts: 2
Member No.: 305
Joined: 16-June 11









I've been trying connect to our Secure NFS implimentation with krb5 1.9 and NFS3 on SL 6 but I consistently get teh error below. This configuration worked fine on SL5.5, and I was able to get it working on FC15, but this keeps failing on the local machine. KRB5.conf and the keytabs appear to be setup correctly, but I still get the following error:

QUOTE
ERROR: GSS-API: error in gss_set_allowable_enctypes(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - Unknown error


Wireshark indicates that it doesn't try to connect.

If anyone can point me in the right direction, or has seen this error before, I'd appreciate your feedback.

Full Log of rpc.gssd -vvvv
CODE

handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clntc)
handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clntc)
process_krb5_upcall: service is '<null>'
Full hostname for 'engfs.eng.ad.ucalgary.ca' is 'engfs.eng.ad.ucalgary.ca'
Full hostname for 'ict21606.eng.ad.ucalgary.ca' is 'ict21606.eng.ad.ucalgary.ca'
No key table entry found for ICT21606$@ENG.AD.UCALGARY.CA while getting keytab entry for 'ICT21606$@ENG.AD.UCALGARY.CA'
No key table entry found for root/ict21606.eng.ad.ucalgary.ca@ENG.AD.UCALGARY.CA while getting keytab entry for 'root/ict21606.eng.ad.ucalgary.ca@ENG.AD.UCALGARY.CA'
Success getting keytab entry for 'nfs/ict21606.eng.ad.ucalgary.ca@ENG.AD.UCALGARY.CA'
INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_ENG.AD.UCALGARY.CA' are good until 1308287718
INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_ENG.AD.UCALGARY.CA' are good until 1308287718
using FILE:/tmp/krb5cc_machine_ENG.AD.UCALGARY.CA as credentials cache for machine creds
using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_ENG.AD.UCALGARY.CA
creating context using fsuid 0 (save_uid 0)
ERROR: GSS-API: error in gss_set_allowable_enctypes(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - Unknown error
WARNING: Failed while limiting krb5 encryption types for user with uid 0
WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_ENG.AD.UCALGARY.CA for server engfs.eng.ad.ucalgary.ca
WARNING: Machine cache is prematurely expired or corrupted trying to recreate cache for server engfs.eng.ad.ucalgary.ca
Full hostname for 'engfs.eng.ad.ucalgary.ca' is 'engfs.eng.ad.ucalgary.ca'
Full hostname for 'ict21606.eng.ad.ucalgary.ca' is 'ict21606.eng.ad.ucalgary.ca'
No key table entry found for ICT21606$@ENG.AD.UCALGARY.CA while getting keytab entry for 'ICT21606$@ENG.AD.UCALGARY.CA'
No key table entry found for root/ict21606.eng.ad.ucalgary.ca@ENG.AD.UCALGARY.CA while getting keytab entry for 'root/ict21606.eng.ad.ucalgary.ca@ENG.AD.UCALGARY.CA'
Success getting keytab entry for 'nfs/ict21606.eng.ad.ucalgary.ca@ENG.AD.UCALGARY.CA'
INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_ENG.AD.UCALGARY.CA' are good until 1308287718
INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_ENG.AD.UCALGARY.CA' are good until 1308287718
using FILE:/tmp/krb5cc_machine_ENG.AD.UCALGARY.CA as credentials cache for machine creds
using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_ENG.AD.UCALGARY.CA
creating context using fsuid 0 (save_uid 0)
ERROR: GSS-API: error in gss_set_allowable_enctypes(): GSS_S_NO_CRED (No credentials were supplied, or the credentials were unavailable or inaccessible) - Unknown error
WARNING: Failed while limiting krb5 encryption types for user with uid 0
WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_ENG.AD.UCALGARY.CA for server engfs.eng.ad.ucalgary.ca
WARNING: Failed to create machine krb5 context with any credentials cache for server engfs.eng.ad.ucalgary.ca
doing error downcall
dir_notify_handler: sig 37 si 0x7fff74a4e530 data 0x7fff74a4e400
dir_notify_handler: sig 37 si 0x7fff74a4e530 data 0x7fff74a4e400
dir_notify_handler: sig 37 si 0x7fff74a4e530 data 0x7fff74a4e400
dir_notify_handler: sig 37 si 0x7fff74a4e530 data 0x7fff74a4e400
dir_notify_handler: sig 37 si 0x7fff74a4e530 data 0x7fff74a4e400
dir_notify_handler: sig 37 si 0x7fff74a4e530 data 0x7fff74a4e400
destroying client /var/lib/nfs/rpc_pipefs/nfs/clntc
PM
^
helikaon
 Posted: Jun 20 2011, 10:29 AM
Quote Post


SLF Moderator
******

Group: Moderators
Posts: 538
Member No.: 4
Joined: 8-April 11









Hi,
just read it, but that made not hint to me so far.
Just few Q for the start - did you check:

- NFS protocol version on server and client/s (version 3 x 4)
- did you check the firewall - ports stated in /etc/sysconfig/nfs (if 'unhashed' any?), portmapper 111 allowed?
- do you have services nfs, nfslock and rpcbind enabled in correct runlevels?

Also can you post the kerberos configs from server and client .... and also ... anything else in logs ... /var/log/messages nothing? /var/log/secure nothing??

The more you post, the better :-)

p.s. did you look e.g.:
secure NFS fedora wiki

cheers,


--------------------
PMEmail Poster
^
sworkhard
 Posted: Jul 4 2011, 05:41 PM
Quote Post


SLF Newbie


Group: Members
Posts: 2
Member No.: 305
Joined: 16-June 11









Thanks. I'll try that guide and see if I can pass that test first.
PM
^
michiel_ph
 Posted: Sep 13 2011, 10:21 AM
Quote Post


SLF Newbie


Group: Members
Posts: 14
Member No.: 833
Joined: 13-September 11









Was this issue resolved somehow? I'm having very similar problems with rpc.gssd and spent the last few days getting it to work properly.
PMEmail Poster
^
helikaon
 Posted: Sep 14 2011, 04:43 AM
Quote Post


SLF Moderator
******

Group: Moderators
Posts: 538
Member No.: 4
Joined: 8-April 11









Hi,
i dont know how this ended for sworkhard, but would be nice to know :-).
Anyway, if you'd describe your environment and problems, then we could help ...
The NFS is sometimes tricky and also! default in rhel 5.5 is NFS v3 and in RHEL 6 is NFS v4, this should be remembered ..
cheers,


--------------------
PMEmail Poster
^
michiel_ph
 Posted: Sep 14 2011, 09:17 AM
Quote Post


SLF Newbie


Group: Members
Posts: 14
Member No.: 833
Joined: 13-September 11









Thanks for the response.

My problem is that rpc.gssd crashes with a segfault. The system is a fresh SL-6.1 install with all updates applied. NFSv4 works perfectly with auth=sys, not functional with the desired kerberos auth.

I can deal with misconfigurations, not with crashes.

My first question is, did this ever work for someone with SL-6.1 or RHEL-6.1 ?
PMEmail Poster
^
helikaon
 Posted: Sep 14 2011, 09:23 AM
Quote Post


SLF Moderator
******

Group: Moderators
Posts: 538
Member No.: 4
Joined: 8-April 11









Hi,
can't help much on your Q. since the NFS v.4 is still a bit new for me.
I can recommend you to use the 'strace' command and see where exactly the binary crashes. It might give us / you lead where to look for the problem ...

cheers,


--------------------
PMEmail Poster
^
michiel_ph
 Posted: Sep 14 2011, 11:50 AM
Quote Post


SLF Newbie


Group: Members
Posts: 14
Member No.: 833
Joined: 13-September 11









Thanks for the advice.

The strace output didn't tell me much, posting the tail of it here just in case someone else understands better.

In the mean time, i'll continue looking..

CODE

[root@us1 ~]# tail -40 /tmp/strace-rpc.gssd
read(13, "\0\f\0\1\0\10\377\377\377\374\0\0\0\0\0\0\0\1\0\0\0\2\0\0\0\16DEV.IN"..., 1024) = 622
lseek(13, -608, SEEK_CUR)               = 16
read(13, "\0\0\0\1\0\0\0\2\0\0\0\16DEV.INOPSY.COM\0\0\0\3nf"..., 1024) = 608
fcntl(13, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
close(13)                               = 0
stat("/etc/krb5.conf", {st_mode=S_IFREG|0644, st_size=481, ...}) = 0
open("/etc/krb5.conf", O_RDONLY)        = 13
fcntl(13, F_SETFD, FD_CLOEXEC)          = 0
fstat(13, {st_mode=S_IFREG|0644, st_size=481, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd11693d000
read(13, "[logging]\n default = FILE:/var/l"..., 4096) = 481
read(13, "", 4096)                      = 0
close(13)                               = 0
munmap(0x7fd11693d000, 4096)            = 0
open("/dev/urandom", O_RDONLY)          = 13
fcntl(13, F_SETFD, FD_CLOEXEC)          = 0
fstat(13, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
read(13, "p\25q\240SA\3560($\4-\331\210\235\t\325G\365N", 20) = 20
close(13)                               = 0
stat("/etc/krb5.conf", {st_mode=S_IFREG|0644, st_size=481, ...}) = 0
open("/etc/krb5.conf", O_RDONLY)        = 13
fcntl(13, F_SETFD, FD_CLOEXEC)          = 0
fstat(13, {st_mode=S_IFREG|0644, st_size=481, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd11693d000
read(13, "[logging]\n default = FILE:/var/l"..., 4096) = 481
read(13, "", 4096)                      = 0
close(13)                               = 0
munmap(0x7fd11693d000, 4096)            = 0
open("/dev/urandom", O_RDONLY)          = 13
fcntl(13, F_SETFD, FD_CLOEXEC)          = 0
fstat(13, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
read(13, "\331(0mRp\302\36*\2025\32t\"e[\220\232\265\306", 20) = 20
close(13)                               = 0
open("/etc/gssapi_mech.conf", O_RDONLY) = 13
fstat(13, {st_mode=S_IFREG|0644, st_size=801, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd11693d000
read(13, "# Example /etc/gssapi_mech.conf "..., 4096) = 801
read(13, "", 4096)                      = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
PMEmail Poster
^
helikaon
 Posted: Sep 14 2011, 02:44 PM
Quote Post


SLF Moderator
******

Group: Moderators
Posts: 538
Member No.: 4
Joined: 8-April 11









QUOTE (michiel_ph @ Sep 14 2011, 11:50 AM)
Thanks for the advice.

The strace output didn't tell me much, posting the tail of it here just in case someone else understands better.....


Hi,
try check the /etc/gssapi_mech.conf and the libraries it mentions, also have a look in whole strace, if any here mentioned library is missing, try look for any other 'error' there etc ...

cheers,


--------------------
PMEmail Poster
^
michiel_ph
 Posted: Sep 14 2011, 03:28 PM
Quote Post


SLF Newbie


Group: Members
Posts: 14
Member No.: 833
Joined: 13-September 11









again, thanks for the help.

I guess, help from the developers is needed. About half the time rpc.gssd crashes, or i get the error mentioned by the original poster. Without support contract with RH, it will be virtually impossible to get a developers attention sad.gif

Apparently NFSv4 with kerberos isn't used anywhere and therefore not tested.

The gssapi_mech.conf chooses between GSSAPI implementations. In my case kerberos is needed. What i understand from the documentation, the other options are certificates.

I had this working with Fedora-13 and -14 at home. That was broken for the same reason with -15. And now, SL seems to have the same problem.
PMEmail Poster
^
helikaon
 Posted: Sep 14 2011, 06:30 PM
Quote Post


SLF Moderator
******

Group: Moderators
Posts: 538
Member No.: 4
Joined: 8-April 11









Hi,
i found this:
Kerberos issues
Check hostnames
Kerberos requires the hostname/domainname used in the keytab is correct. Run `hostname` and look in /etc/hosts to doublecheck that it is set properly. Compare with what you've listed in your keytab file.

Check keytabs
Run the following command to check your keytab:
CODE

klist -k

Check krb5 ccache file
If you see log messages regarding something like 'FILE:/tmp/krb5cc_machine_FOO.BAR.AD.ROOT', you can review the file after trying to do the mount via:
CODE

klist -e -f -c /tmp/krb5cc_machine_FOO.BAR.AD.ROOT

This will list info about your principals such as the valid/expire dates, encryption types, etc.

this is found at nfs wiki:
NFS wiki

cheers,



--------------------
PMEmail Poster
^
zzxtty
 Posted: Sep 15 2011, 09:26 AM
Quote Post


SLF Newbie


Group: Members
Posts: 7
Member No.: 322
Joined: 21-June 11









I have similar problems with kerberos/nfs not working under 6.1. rpc.gssd will crash for me under certain circumstances, it appears to relate to the number of '-v' I use, '-vv' usually works, '-vvv' usually crashes.

I've checked the /etc/gssapi_mech.conf files on a working 6.0 machine vs a problematic 6.1 machine, they are the same.

On a working 6.0 machine I have the following tickets:
CODE

[root@cnslnx4 ~]# klist -e -f -c /tmp/krb5cc_machine_IOP.KCL.AC.UK
Ticket cache: FILE:/tmp/krb5cc_machine_IOP.KCL.AC.UK
Default principal: root/cnslnx4.iop.kcl.ac.uk@IOP.KCL.AC.UK

Valid starting     Expires            Service principal
09/15/11 09:57:47  09/16/11 09:57:47  krbtgt/IOP.KCL.AC.UK@IOP.KCL.AC.UK
renew until 12/14/11 08:57:47, Flags: FRIA
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
09/15/11 09:57:47  09/16/11 09:57:47  nfs/hawker.iop.kcl.ac.uk@IOP.KCL.AC.UK
renew until 09/15/11 09:57:47, Flags: FRAT
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
[root@cnslnx4 ~]#


On the 6.1 machine:
CODE

[root@msclnx1 ~]# klist -e -f -c /tmp/krb5cc_machine_IOP.KCL.AC.UK
Ticket cache: FILE:/tmp/krb5cc_machine_IOP.KCL.AC.UK
Default principal: root/msclnx1.iop.kcl.ac.uk@IOP.KCL.AC.UK

Valid starting     Expires            Service principal
09/15/11 10:10:32  09/16/11 10:10:32  krbtgt/IOP.KCL.AC.UK@IOP.KCL.AC.UK
renew until 12/14/11 09:10:32, Flags: FRIA
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
[root@msclnx1 ~]#


This tells me that the client machine has failed to get a ticket from the disk server (hawker), but my knowledge of kerberos is limited. It's something I'm trying to get working but haven't been able to get very far, SL6.0 was the only version that has shown any promise of actually working. I suppose this raises the question, if kerberos/nfs doesn't work, how do people authenticate their NFS clients?
PMEmail Poster
^
michiel_ph
 Posted: Sep 15 2011, 09:30 AM
Quote Post


SLF Newbie


Group: Members
Posts: 14
Member No.: 833
Joined: 13-September 11









helikaon: Very useful tips indeed.

I wonder whether SL-6.1 supports the stronger encryptions for NFS. In various places i saw instructions to use DES only for NFS. Other places mentions newer kernels and/or distros support the stronger encryptions like AES.

Using google, the closest match to our problems seems to be: http://permalink.gmane.org/gmane.linux.nfs/41950 .

What a mess...

Anyway, thanks for the help so far.


Mvg,
Michiel
PMEmail Poster
^
michiel_ph
 Posted: Feb 18 2012, 06:04 PM
Quote Post


SLF Newbie


Group: Members
Posts: 14
Member No.: 833
Joined: 13-September 11









With SL-6.2, the problems have disappeared.
PMEmail Poster
^
0 User(s) are reading this topic (0 Guests and 0 Anonymous Users)
0 Members:

Topic Options Reply to this topicStart new topicStart Poll