Troubleshooting Project Issues

This is a guide on troubleshooting some common issues with Anaconda Enterprise notebook projects. Issues can range from project creation, startup, collaboration, and individual applications in the project.

Project Creation/Startup Issues



If project creation hangs for a long time or eventually gives an error the best place to look first is in the compute-launcher.application.log located in /opt/wakari/wakari-compute/var/log/wakari/.
 - NFS related error:

Creating directory /projects/testuser
Setting ACL permission of /projects/testuser to rwx for testuser (testuser)
setfacl: /projects/testuser: Operation not supported

One common issue can be a failure to create a project on an NFS mounted /projects directory. The above error happens when NFS version 4 is being used. since Anaconda Enterprise Notebooks uses ACLs to apply permissions to projects NFS version 3 is a requirement.
    - New User Issues:

sudo: unable to change directory to /home/testuser: No such file or directory
sudo: unable to execute /bin/bash: No such file or directory
The 'wk-app-runner' program exited without launching the application

The above error is related to user management on the Operating System. When AEN is not managing users on the Operating System(by setting MANAGE_ACCOUNTS to false in /opt/wakari/wakari-compute/etc/wakari/config.json) your IT department must create and manage all user accounts at the OS level.

If this occurs with MANAGE_ACCOUNTS set to true you will need to check if the user experiencing the issue already had an account on the server. If an account already exists AEN expects the user's home directory to exist and will not try to create it. You can either create the user's home directory or delete the user from the Operating System so that AEN can manage the creation.

Collaboration/Application Access Issues

Some Applications may issues with permissions. The first place to check is the individual application log file. The are located in the following directory for each owner, project, and user in the following format:

/opt/wakari/wakari-compute/var/log/wakari/compute-launcher-apps/<OWNER>/<PROJECT>/<USER>

Access and permissions are set using ACLs. You can list the users that have permissions and what permissions they have using the following command pointing to a resource or directory:

getfacl /projects/<Owner/<PROJECT>/

    - ACL Issues:

File "/opt/wakari/wakari-compute/lib/python2.7/site-packages/workbench/permissions_manager.py", line 165, in _get_name_for_entry
    return pwd.getpwuid(entry.qualifier)[0], posix1e.ACL_USER
KeyError: 'getpwuid(): uid not found: 503'

When a user has been removed from the Operating System or has had their UID changed the ACLs may list just a UID rather than username. This can cause an iossue as AEN will not be able to find this user. You should remove the UID as it shows up(in the error above it is UID 503).

setfacl -x u:<UID> /path/to/resource

 - Incorrect Permissions/No Permissions:

bash: /projects/testuser/testproject/.projectrc: Permission denied

The above error is an example of not having permissions on a project terminal. This general issue can happen on many application with a basic "Permission denied". You should check the ACLs and compare them to a user that works.

 - Bad Path to Default Environment:

If you are experiencing issues getting an environment or project to work at all you should check the Compute Resource Config application. In the upper right there is a field for the default environment that the project should run in. Be sure it shows  the default path(example: /projects/username/projectname/envs/default), if not you can set it and select "Set Project Environment" then reopen the project. If the default environment is still having issues you may want to follow the article on Rebuilding a Project's Default Environment.