Department of Physics and Astronomy

The Forbes Group

Mac OS X

$\newcommand{\vect}[1]{\mathbf{#1}} \newcommand{\uvect}[1]{\hat{#1}} \newcommand{\abs}[1]{\lvert#1\rvert} \newcommand{\norm}[1]{\lVert#1\rVert} \newcommand{\I}{\mathrm{i}} \newcommand{\ket}[1]{\left|#1\right\rangle} \newcommand{\bra}[1]{\left\langle#1\right|} \newcommand{\braket}[1]{\langle#1\rangle} \newcommand{\op}[1]{\mathbf{#1}} \newcommand{\mat}[1]{\mathbf{#1}} \newcommand{\d}{\mathrm{d}} \newcommand{\pdiff}[3][]{\frac{\partial^{#1} #2}{\partial {#3}^{#1}}} \newcommand{\diff}[3][]{\frac{\d^{#1} #2}{\d {#3}^{#1}}} \newcommand{\ddiff}[3][]{\frac{\delta^{#1} #2}{\delta {#3}^{#1}}} \DeclareMathOperator{\erf}{erf} \DeclareMathOperator{\Tr}{Tr} \DeclareMathOperator{\order}{O} \DeclareMathOperator{\diag}{diag} \DeclareMathOperator{\sgn}{sgn} \DeclareMathOperator{\sech}{sech} $

Table of Contents

Here are some notes about how I configure Mac OS X.

MacPorts

MacPorts brings to Mac OS X many programs available on Linux. The programs are compiled from source, which means it can take a while to install everything and that they need the Xcode command line tools installed:

These are installed in /opt/local/ and to use them, you should add /opt/local/bin to your initialization scripts. (I do this with various configuration files in my configurations project: see for example my ~/.environment_site file.)

I do not like running these commands as root, so once I install MacPorts, the first thing I do is make everything owned by an unprivledged user (me).

sudo find /opt/local -exec chown mforbes {} \;

This has a couple of complications (see rsync below) but works well for the most part. If you do this, be sure to run port without sudo which will be counter to the usual instructions found on the internet. If there is a permissions issue, then check where and deal with it on a case-by-case basis.

Alternatives

  • Homebrew provides a popular alternative. From what I can gather from various discussions, MacPorts is more stable, better designed, and more reliable. Some key points are:
    • MacPorts installs in /opt/local. In contrast, Homebrew puts everything in /usr/local while also changing the permissions of everything in there to the single user running. This might make sense with a single user – and indeed is similar to what I do with permissions for /opt/local, but apparently makes it much more difficult for multi-user machines. It also presents a security hazard and is bad practice. For me, this is an issue, because I expect everything in /usr/local to be customize by me – stuff I install from source for example.
    • MacPorts installs redundant but isolated libraries, so does not break when Apple updates things. In contrast, Homebrew uses system libraries. Makes it smaller, but less robust.

Discussions

Useful Commands

See Common Tasks for more details.

  • Information:
port outdated
port installed inactive
port installed requested
port echo leaves
port dependents <portname>
  • Marking ports as requested (so they don't appear as leaves)
port setrequested ...
  • Updating and Cleaning
port selfupdate
port upgrade outdated
port uninstall inactive
port uninstall leaves         # Removes one level of leaves
port uninstall rleaves        # Removes all leaves (unrequested ports)

Here is what I typically install:

port install aspell aspell-dict-en 
port install ffmpeg +nonfree ImageMagick pngcrush
port install coreutils fswatch gawk wget tree shellcheck ncdu textbrowser
port install gt5
port install cvs bzr git myrepos
port install gsl
port install cmake gmake gperf swig-python
port install symlinks
port install rmtrash
port install munin
port install lua luarocks    # For Lmod: see below
port install jq              # JSON parser - used for migrating to hg.src.ht
port install multimarkdown
port install py-altgraph graphviz   # For visualizing dependencies
port install sshuttle  # Tool for simple ssh VPN.
port install smartmontools  # Tools for monitoring harddive performance etc.

The rsync package needs access to /Library/LaunchDaemons.

sudo chmod a+w /Library/LaunchDaemons
port install rsync
sudo chmod a-w /Library/LaunchDaemons

Cleaning

To clean your installation you might want to do the following:

port clean --all all        # Can be very slow...
port uninstall inactive
port uninstall rleaves

According to this answer, you might also be able to delete the following:

rm -rf /opt/local/var/macports/software

After installation, my typical setup consumes about 1.7GB of disk space – most of which is due to ffmpeg:

$ du -sh /opt/local/
1.7G    /opt/local/

Dependencies

In [ ]:
!curl -O https://raw.githubusercontent.com/Synss/macports_deptree/master/port_deptree.py
!pip install --user altgraph
!/opt/local/bin/python27 port_deptree.py python27 | dot -Tpdf | open -fa Preview
#!python port_deptree.py py-altgraph | dot -Tpdf | open -fa Preview
#!python port_deptree.py ffmpeg +nonfree | dot -Tpdf | open -fa Preview

Custom Portfiles

If you need to create your own Portfile or slightly modify an existing one you can quite easily by creating your own Local Portfile Repositories. Here is how I did this using a local repository in ~/src/ports to edit the sshuttle Portfile to depend on Python 3.9 (so MacPorts only brings in one version of python).

  1. Add file:///Users/mforbes/src/ports to /opt/local/etc/macports/sources.conf.

    conf
# /opt/local/etc/macports/sources.conf
...
file:///Users/mforbes/src/ports
rsync://rsync.macports.org/release/tarballs/ports.tar [default]

  2. Create the Portfile:

    mkdir -p /Users/mforbes/src/ports/net/sshuttle
curl https://raw.githubusercontent.com/macports/macports-ports/master/net/sshuttle/Portfile > /Users/mforbes/src/ports/net/sshuttle/Portfile

  3. Edit the Portfile:

    #/Users/mforbes/src/ports/net/sshuttle/Portfile
...
python.default_version 39
...

  4. Update the index:

    cd /Users/mforbes/src/ports
portindex

  5. Update the port and clean:

    port uninstall sshuttle   # Get rid of old version
port search sshuttle
port install sshuttle
port uninstall inactive
port uninstall leaves
port uninstall rleaves

References

Shell (bash)

Sometimes one might like to use another shell instead of /bin/bash. For example, one might like to use the version /opt/local/bin/bash provided by MacPorts so that one can use the bash-completion package. To do this, we first install the shell, then enable it, finally we change the user shell:

port install bash-completion
sudo echo /opt/local/bin/bash >> /etc/shells
chsh -s /opt/local/bin/bash
mkdir -p ~/.local/share/bash-completion/completions

The for completions, I do things like:

poetry completions bash > ~/.local/share/bash-completion/completions/poetry.bash

Finally, in my ~/.bashrc file, I have:

#~/.bashrc
...
if [ -f /opt/local/etc/profile.d/bash_completion.sh ]; then
  . /opt/local/etc/profile.d/bash_completion.sh
fi

XCode

XCode is huge (~10GB), so I remove it (just drag XCode.app to the trash) and just keep the command line tools installed:

xcode-select --install

After agreeing to the licence, you should see

$ xcode-select -p
/Library/Developer/CommandLineTools

Note: after you do this you will likely see the following warning from MacPorts:

* `Warning: xcodebuild exists but failed to execute`
* `Warning: All compilers are either blacklisted or unavailable; defaulting to first fallback option`

Conda

In install Conda with the following in mind:

  1. I use Miniconda as opposed to the full Anaconda distribution. If I need Anaconda, then I install it in a special environment.
  2. I install Conda as a special user conda so that I can't accidentally muck up the environments. This simulates what happens on HPC compute clusters etc. where we share conda environments.
  3. Environments are specified with environment.yml files and Picky for Conda should be used to lock these.
  4. Custom work should be done in a virtual environment that sits on one of these base conda environments.

Here is the complete setup process:

  1. Create a conda user account. (Probably easiest just to use the GUI.)

  2. I was going to enable passwordless su access, but it seems easier just to enable an ssh alias:

    #~/.ssh/config
...
Host conda_local
  User conda

Host admin_local
  User admin

Host *_local
  Hostname localhost
  ForwardAgent yes

    Now copy the keys:

    ssh-copy-id conda_local

  3. Create the base environments:

    sudo mkdir -p /data/apps/conda    # Remove an old installation if needed
sudo chown conda /data/apps/conda
ssh conda
mkdir -p zips
cd ~/zips
curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O
bash Miniconda3-latest-MacOSX-x86_64.sh -bfp /data/apps/conda
/data/apps/conda/bin/conda init
. ~/.bash_profile

# Update the base environment with anaconda-client, then from my anaconda page:
# https://anaconda.org/mforbes/base
/data/apps/conda/bin/conda install anaconda-client -n base
/data/apps/conda/bin/conda update -n base mforbes/base
echo 'eval "$(mmf_setup -v -H)"' >> ~/.bash_profile

# Added some bare python environments.  I use these for testing with Nox.
for py in 3.6 3.7 3.8 3.9; do
  /data/apps/conda/bin/conda create -y -c defaults --override-channels -n "py${py}" python=${py}
done

  4. Create some specialized environments. Ideally these should be able to be created from my anaconda channel:

    ssh conda
conda env update mforbes/jupyter
conda env create mforbes/work

    If you are updating them, it is useful to have them locally:

    ssh conda
mkdir -p repositories
cd repositories
hg clone ssh://git@hg.iscimath.org:11022/mforbes/configurations
conda env create

Networking

To see your MAC address:

ifconfig en0 ether

If you need to change it, then you can spoof it with

sudo ifconfig en0 ether xx:xx:xx:xx:xx:xx

Note: make sure that en0 is the correct device. You might want to check the ouput of ifconfig first to see that this is indeed the active device.

VPN

To connect to various services at WSU, I downloaded their Global Protect client. This is needed for some reason. In particular, without using the VPN, the following does not work:

(swan) $ nc -l 12345   # Start a netcat server listing on port 12345 on swan

then on my Mac:

(Rentdem) $ nc swan.physics.wsu.edu 12345   # Start a netcat client

Without VPN, I can only send about 4 messages in either direction, then the connection hangs. Not sure why.

Wireshark

Wireshark is a full-featured debugging tool for networks. I install it and add it to the PATH so I can use it in a terminal.

SSH

To tell KeyChain to remember your SSH keys do the following:

ssh-add -K [path/to/private SSH key]

SSHFS

You can use SSH to mount remote file systems locally with sshfs. On Mac OS X, do this by installing FUSE and the associated SSHFS package from that site. Once it is installed, you can mount a filesystem as follows:

mkdir -p ~/Volumes/swan
sshfs -o follow_symlinks swan:/ ~/Volumes/swan

To unmount:

umount ~/Volumes/swan

SSH Tunnels (SOCKS/VPN) 

sshuttle user@host 0.0.0.0/0 -vv

This will forward all traffic through the specified host. See https://superuser.com/a/757974. sshuttle can be installed with pip or macports.

BBCP

Related to SCP is bbcp which can be downloaded for OS X here. This is a fast tool for transferring files. I just copy it to /usr/local/bin/bbcp.

LaTeX

I install the MacTeX which puts most things in /usr/local/texlive and also includes BibDesk, Skim etc. When you upgrade from one year to the next (say 2017 to 2018), the new installation will be placed in /usr/local/texlive/2018 leaving the old version in /usr/local/texlive/2017. These can be big (~5.4GB) so you might want to move the old version off your hard-drive (but keep it for a bit in case things go wrong.)

Once you upgrade, you may need to make a few changes to keep things working, setting the path for the LaTeX programs. These should be set to

/Library/TeX/texbin

but might need to be updated. Check in the preferences of the following packages if things do not work as expected:

  • BibDesk: (Seems okay with symlink above.)
  • TeX Live: (Needed path refreshed as it seems to use the absolute path, but can refresh this for you if you confirm.)
  • LaTeXiT: (Seems okay with symlinks.)

Safari

Extensions

  • Ad Blocking: There are several extensions that block ads.

    • AdBlock: I have been using this and am reasonably happy with it. The blocking of ads in YouTube is especially valuable.
    • AdBlock Plus: Another alternative that is almost the same (small UI differences).
    • AdGuard for Safari: Was recommended in an uBlock issue about Safari. Does not block YouTube ads, but I have not tried the full AdGuard (just the browser extension).

    For a comparison of the first two, see AdBlock vs AdBlock Plus.

  • StopTheMadness: Some websites disable useful features such as copy and paste. This extension claims to restore this functionality.

    I used a different approach - temporarily disable JavaScript. This can be done under the Safari Develop/Disable JavaScript menu item, but I also enabled a shortcut ⌘J to Disable JavaScript..

Emacs

There are several options for using Emacs on Mac OS X:

  • GNU Emacs for Mac OS X: I use this. It is a fairly generic version of emacs with traditional key bindings. The only annoyance is that one must use Esc for the Meta key. (M-x is achieved by "pressing and releasing Esc, then pressing x.)
  • Aquamacs: This more tightly integrates with the OS X interface, but is significantly slower.

Configuration

I install the following packages using the Options/Manage Emacs Packages menu option. Note, there are three main package archives. I use these with the following preference: 1) gnu, 2) marmalade, 3) melpa (see this discussion for more details.

Editing Features

  • autopair: Automagically pair braces and quotes like TextMate.
  • org: Outline-based notes management and organizer.

Modes:

  • auctex-latexmk: Add LatexMk support to AUCTeX.
  • haskell-mode: A Haskell editing mode.
  • lua-mode: A major-mode for editing Lua scripts.
  • markdown-mode+: Extra functions for markdown-mode.
  • markdown-preview-mode: Markdown realtime preview minor mode.
  • markdown-toc: A simple TOC generator for markdown file.
  • yaml-mode: Major mode for editing YAML files

Elisp Debugging (init files)

  • benchmark-init: Benchmarks Emacs require and load calls.
  • bug-hunter: Hunt down errors by bisecting elisp files.
  • use-package: A configuration macro for simplifying your .emacs.

Git: (I don't really use these, but they were recommended at some point.)

  • git-commit-mode: Major mode for editing git commit messages [github].
  • git-rebase-mode: Major mode for editing git rebase files [github].
  • magit: A Git porcelain inside Emacs.

Python: Modes for working with python.

  • conda: Work with your conda environments
  • python-mode: Python major mode. This has some major problems when using Tramp for remote editing, so I do not use it any more.
  • elpy: Emacs Python Development Environment

Web/Blog:

  • sass-mode: Major mode for editing Sass files.
  • scss-mode: Major mode for editing SCSS files.
  • json-mode: Major mode for editing JSON files.
  • jinja2-mode: A major mode for jinja2.
  • tidy: Interface to the HTML Tidy program.

I load these in my .emacs file with the following code

;; Make sure required packages are installed
(setq package-list
      '(use-package
    autopair
    org
    auctex-latexmk
    haskell-mode
    lua-mode
    markdown-mode+
    markdown-preview-mode
    markdown-toc
    yaml-mode
    benchmark-init
    bug-hunter
    use-package
    ;git-commit-mode
    ;git-rebase-mode
    ;magit
    conda
    ;python-mode
    elpy
    sass-mode
    scss-mode
    json-mode
    jinja2-mode
    tidy))

;; list the repositories containing them
(setq package-archives
      '(
    ("gnu" . "http://elpa.gnu.org/packages/")
    ("marmalade" . "http://marmalade-repo.org/packages/")
    ("melpa" . "http://melpa.milkbox.net/packages/")
    ;("elpa" . "http://tromey.com/elpa/")
    ))

;; activate all the packages (in particular autoloads)
(package-initialize)

;l fetch the list of packages available 
(unless package-archive-contents
  (package-refresh-contents))

;; install the missing packages
(dolist (package package-list)
  (unless (package-installed-p package)
    (package-install package)))

iPhone/iPad

Backups

If you make backups of your iPhone or iPad with iTunes, you can locate these with

  • iTunes/Preferences/Devices/Device Backups and right-click to Reveal in Finder.

These files, however, are not organized and obfuscated. To interpret this data you seem to need to use a third party application. Most of these have a demo mode that will allow you to view the files, but put severe limitations on how much data you can actually extract without paying.

Useful Applications

GPG

This is useful for encryption and signing files. I use the vi plugin which allows you to edit encrypted files.

Preference Panes

  • Choosy: A "browser" that redirects websites to specific browsers. I am just trying this, but hope to use it to open CoCalc websites exclusively in Chrome for example even though I generally prefer Safari.

Vagrant

Vagrant provides a platform for running code in isolated virtual environments. If you need more fine-grained control and provisioning, [Docker] is another option, but I use Vagrant for developing websites for example.

Docker/AWS-CLI

I don't really use Docker on my Mac, but install it so I can provision images on AWS. To do this I did the following:

  1. Make a directory for installing the aws-cli without sudo:

    sudo mkdir /usr/local/aws-cli/
sudo chown mforbes /usr/local/aws-cli/

  2. Download and install the aws-cli as a local user ("Install for Me Only").

    $ ln -s /usr/local/aws-cli/aws /usr/local/aws-cli/aws_completer /usr/local/bin/
$ aws --version
aws-cli/2.1.10 Python/3.7.4 Darwin/18.7.0 exe/x86_64 prompt/off

  3. Install the Docker App for OS X. Note: I though I could use port install docker docker-machine and save on some disk space, but this needs Vagrant, so no clear savings there. There are some relevant discussions in this direction though:

Disk Space:

Docker uses lots of space. Some can be reclaimed with:

docker images
docker rmi <images>
docker system prune
docker run --privileged --pid=host docker/desktop-reclaim-space

Environment Modules (Lmod)

To maintain some coherence with HPC environments, we install some version of the Environment Modules package. Here we use the Lmod variant as this is used on our local cluster.

luarocks install luaposix
luarocks install luafilesystem
LUAROCKS_PREFIX=/opt/local/share/luarocks
export LUA_PATH="$LUAROCKS_PREFIX/share/lua/5.3/?.lua;$LUAROCKS_PREFIX/share/lua/5.3/?/init.lua;;"
export LUA_CPATH="$LUAROCKS_PREFIX/lib/lua/5.3/?.so;;"


APP="Lmod"
VER="8.4"
NAME="${APP}-${VER}"
FILE="${NAME}".tar.bz2
cd ~/zips/
wget https://sourceforge.net/projects/lmod/files/"${FILE}"
cd ~/src/
tar -jxvf ~/zips/"${FILE}"
cd "${NAME}"
./configure --prefix=/data/apps/
make install

I then added the following (from Environment Modules):

cat << EOF > /data/apps/lmod/lmod/modulefiles/Core/use.own
#%Module1.0#####################################################################
##
## use.own modulefile
##
proc ModulesHelp { } {
    puts stderr "\tThis module file will add \$HOME/privatemodules to the"
    puts stderr "\tlist of directories that the module command will search"
    puts stderr "\tfor modules.  Place your own module files here."
    puts stderr "\tThis module, when loaded, will create this directory"
    puts stderr "\tif necessary."
}

module-whatis   "adds your own modulefiles directory to MODULEPATH"

eval set  [ array get env HOME ]
set ownmoddir   $HOME/privatemodules

# create directory if necessary
if [ module-info mode load ] {
    if { ! [ file exists $ownmoddir ] } {
        file mkdir $ownmoddir
        set null [open $ownmoddir/null w]
        puts $null "#%Module########################################################################"
        puts $null "##"
        puts $null "## null modulefile"
        puts $null "##"
        puts $null "proc ModulesHelp { } {"
        puts $null "    puts stderr \"\tThis module does absolutely nothing.\""
        puts $null "    puts stderr \"\tIt's meant simply as a place holder in your\""
        puts $null "    puts stderr \"\tdot file initialization.\""
        puts $null "}"
        puts $null ""
        puts $null "module-whatis   \"does absolutely nothing\""
    }
}

module use --append $ownmoddir
EOF

To use these, I add the following to my ~/.environment_site file:

# Site specific bash environment init file; -*-Shell-script-*-
# dest = ~/.environment_site #### Keep this as the 2nd line for mmf_init_setup

# This file is loaded by .environment and should be used to set the
# environment for site-specific customizations

...

test -f "/data/apps/lmod/lmod/init/profile" \
  && . "/data/apps/lmod/lmod/init/profile"

...
# Load useful modules.
module load use.own cuda git-annex node mongodb fftw # anaconda

For details about what these do, see my modules configurations folder.

Graphics

MacSVG

Program for producing SVG graphics.

GIMP

Opensource replacement for Photoshop.

CMake

Needed by some applications (I needed it to build VisIt plugins. I simply added this line to my environment_site file:

prepend_path PATH "/Applications/CMake.app/Contents/bin"

Julia

Some tools - in particular the diffeq library - use Julia, but it is not so easy to install via Conda. I install the native application. After installing I had to link it:

ln -s /Applications/Julia-1.5.app/Contents/Resources/julia/bin/julia ~/.local/bin/
conda activate jupyter
export JUPYTER="$(type -p jupyter)"
julia

Then, using julia:

using Pkg
Pkg.add("IJulia")
Pkg.build("IJulia")

Note: If this fails, you might need to manually clone the registry:

git clone https://github.com/JuliaRegistries/General.git ~/.julia/registries/General

Then, in python, I needed to:

conda activate work
pip install diffeqpy
python -c "import diffeq;diffeq.install()"

This installed everything needed into ~/.julia.

References

Sound: Black Hole

I used to use SoundFlower to route audio on my Mac. This is no longer supported, but they recommend Loopback. Another option is BlackHole as discussed in this article. JACK is another option (this is used by Audacity – a great open-source tool for editing audio.)

LICEcap

Simple program for capturing activity on your screen and turning it into an animated GIF. Useful for demonstrating how to use an interface.

Sim Datlonism

Color-blind simulator. Useful for checking graphs, posters, etc.

Microsoft Office

I have the problem of running with fairly limited disk space, so installing the complete Office suite was prohibitive. For a long time I stuck with Microsoft Word for Mac 2011, which I got from the department. It appears now that individual apps are available through the App Store:

In [ ]: