DreamTeam/Valis

From Noisebridge
Revision as of 20:25, 16 January 2021 by Bfb (talk | contribs) (→‎Services)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

About[edit | edit source]

Valis is a shared compute resource for training machine learning models. Specifically, Valis excels at tasks that require GPU compute. Valis is intended for research and educational purposes only (not for profit).

Specs[edit | edit source]

  • Intel 4114 10-core CPU
  • 4x Nvidia Titan V GPUs
  • 196GB ECC RAM
  • 2x 2TB SSDs

Access[edit | edit source]

Valis is accessed through a VPN. Follow the instructions below to receive and utilize a VPN user account.

  1. Let the community know about your project by sending a message to the ml list or join the Dream Team Neurohackers in person on Wednesday nights at 8pm.
  2. Make a note on Talk:Valis with your GPG public key fingerprint. You will be emailed an encrypted archive containing your cert, key, OpenVPN configuration file, and OpenVPN username/password for connecting to the VPN plus a Jupyterhub username/password.
    1. Jupyterhub and OpenVPN currently use two different user access systems.
  3. Configure your computer to connect to our OpenVPN server
    1. On Debian this can be configured using network-manager-openvpn.
    2. On OS X Tunnelblick works well.
  4. Connect to the VPN and run #verifications.
  5. If the verifications pass, then continue. If not, see #Debugging/FAQ.
  6. Change your VPN password here.
    1. Please note that all TLS certs on the internal network are signed by an internal CA. Your browser will not trust this CA, nor the certs that it has signed by default.
  7. Login to our Jupyterhub instance.
  8. Open a terminal session and #change your linux user password.
  9. Open a Python 3 notebook and begin hacking.

verifications[edit | edit source]

$ ping 172.16.1.1
$ ping 172.16.3.12
$ ping valis

change your linux user password[edit | edit source]

$ passwd

Scheduling[edit | edit source]

Notes on how/what to check before running a process


Debugging/FAQ[edit | edit source]

Services[edit | edit source]

URL Comments
https://pfsense/ Firewall, DNS, OpenVPN server, etc.
https://valis:7999/ Notebook server
https://valis:9999/ Metrics dashboards

Installing New Python Packages[edit | edit source]

  1. Activate the venv source /usr/local/share/pyvenvs/jupyterhub/bin/activate
  2. pip install [name-of-package]
  3. Leave venv deactivate

Users by default have read-only access to the filesystem

If there's an EnvironmentError during the pip install recursively set the file group to jupytervenv

chown -R :jupytervenv /usr/local/share/pyvenvs/jupyterhub/

This should fix things for linux users who are members to the jupytervenv group. There's probably a better way for multiple linux users to share/maintain a python virtual environment.

(jupyterhub) kjs@valis:~$ cat /etc/group | grep jupytervenv jupytervenv:x:1005:kjs,nick,jade,fineline,noisebridge (Feb 2019)