Managing Multiple AWS and Git profiles

Being a machine learning engineer is tough nowadays. In addition to managing machine learning pipeline config, we also have to juggle between multiple git profiles and AWS accounts, hoping not to start an expensive training job for work on the personal account, or committing code to work repo using personal git profile. The following will help managing and toggle between different profiles.

1. Managing Multiple Git Profiles

We can structure .gitconfig as the following to handle multiple profiles. The trick is to let git choose which .conf file to use for different directories. For example, in the following sample .gitconfig, the default config is git-personal.conf. And if we execute git command under /usr/work_directory, git will automatically use git-work.conf as its configuration.

[include]
    path = ~/git-personal.conf
[includeIf "gitdir:/usr/work_directory/"]
    path = ~/git-work.conf

Here’s a sample git-personal.conf:

[user]
    email = snorlax@home.com
    name = slacking
[credential]
    helper = manager
[core]
    autocrlf = true

And here’s a sample git-work.conf:

[user]
    email = serious.mode@work.com
    name = han lee
[credential]
    helper = manager
[core]
    autocrlf = true

We can extend this setup to multiple different git accounts and directories. In our experience, this config-file-based setup is easier and more flexible than the equivalent of SSH-based git profile management.

2. Managing multiple AWS CLI Credentials

For AWS CLI Credentials, we can again set different profiles for usage. The config file is usually located at ~/.aws/config and the credentials are usually stored in ~/.aws/credentials. For example, we can set the default region to us-east-1 and personal profile region to us-west-2 in config as follows:

[default]
region = us-east-1
[profile personal]
region = us-west-2

Then, we can set the respective access key and secret access key as:

[default]
aws_access_key_id = AAAAAAAAAAACESSKEYID
aws_secret_access_key = asdfg+werwjerwg234jg324gU34g+k/secretKEY
[personal]
aws_access_key_id = AAAAAAAAAAACESSKEYID
aws_secret_access_key = asdfg+werwjerwg234jg324gU34g+k/secretKEY

Then, we can toggle between different accounts by either

export AWS_PROFILE=default

And on Windows, run the following and restart the terminal.

setx AWS_PROFILE personal

And to view current profile:

aws configure list

References:

1. Github Account SSH

2. Named profiles

---

To cite this content, please use:

@article{
    leehanchung,
    author = {Lee, Hanchung},
    title = {Feature Selection and Dimensionality Reduction},
    year = {2021},
    howpublished = {\url{https://leehanchung.github.io}},
    url = {https://leehanchung.github.io/2021-04-16-Credential-Management/}
}